acceptodds
Under review as a conference paper at ICLR 2027

Geo-REPA: Geometry-Aware Representation Alignment Enhances Native Spatial Intelligence in MLLMs

Abstract

Multi-modal Large Language Models (MLLMs) excel at semantic understanding but often struggle to reason about physical size, depth, and spatial relationships in 3D environments. Visual geometry encoders provide strong geometric priors, yet our analysis of frozen checkpoints on held-out samples finds that correspondence between MLLM hidden states and geometry features weakens in later layers after input-level fusion, which lacks direct intermediate supervision. We introduce Geo-REPA, a geometry-aware representation alignment framework that encourages intermediate MLLM features to preserve and use 3D cues. To account for camera motion between image and 3D scene coordinate frames, Geo-REPA introduces a pose-aware SE(3)-equivariant inductive bias. Learned pose tokens predict per-frame rotations and translations, which induce a transformation of intermediate visual tokens. A dual alignment objective supervises the transformed visual tokens with 3D geometry features and the pose tokens with camera pose targets from a visual geometry encoder. Across VSI-Bench, SQA3D, and Scan2Cap, Geo-REPA achieves competitive or leading spatial performance, with Scan2Cap gains extending to a newer MLLM backbone. Representation diagnostics associate stronger alignment with lower depth error, while cross-view visualizations show reduced separation among views after transformation. Quantitative analyses associate alignment with spatial performance, show pose sensitivity and meaningful rotation and translation direction estimates. Matched ablations favor pose-aware alignment over no alignment and direct matching across three tasks. On general multimodal benchmarks, the evaluated variant scores modestly below its original backbone. Neither the external geometry encoder nor the alignment modules are needed at inference. Code, scripts, and checkpoints will be released.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation alignment, spatial intelligence, SE(3)-equivariance

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_CPiQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Authors propose to improve the spatial reasoning of MLLMs by applying representational alignment from VGGT to the hidden vision features of an MLLM. They first apply SFT and then do GRPO and observe substantial improvements. The core insight is to apply a 3D transformation to the vision features before alignment, which authors argue drives the bulk of their improvements.

Critical strengths

The experiments are relatively solid. With the exeption of the baseline highlighted in "Questions," I feel that the experiments are relatively solid and show the utility of the method.

Also, the paper is somewhat easy to follow. While I believe some improvements to clarify could be made, there is a clear flow and it can be understood with relatively little "effort".

Critical weaknesses

Incremental novelty and lack of discussion about related works:

First, this work feels relatively incremental. I feel it is somewhat obvious (given that REPA and similar methods have been around for 2 years now), that if we improve the visual representation within a language model for specific tasks (i.e. 3D), we will get better results on related tasks.

A core work that authors appear not to have cited or discussed is Yoon et al, 2025. This work introduces REPA for multimodal LLMs (using the original DINOv2) and shows solid increases in perceptual tasks. I feel that it is conceptually similar to the core insight of this paper, and should be adequately discussed before publication.

Visual Representation Alignment for Multimodal Large Language Models. Heeji Yoon et al. arXiv 2025.

Questions for authors
  1. In my opinion a core reason that MLLMs are interesting is they are used for **general purpose vision tasks. ** However, the approach presented in the paper basically goes in the opposite direction, to finetune an MLLM to be useful for primarily spatial tasks while disregarding many other important visual tasks. Does the SE(3) alignment come at a significant cost for other vision tasks (i.e. chart understanding, pointing, etc)? In this case, it may simply be improving the task-specific representation while degrading general-purpose representations.

  2. Authors first do an SFT stage and then a GRPO stage. If we do a FLOP-for-FLOP analysis, how much (if at all) does the GRPO outperform a continued SFT?

  3. In my opinion, the fair comparison is not comparing to the base model, but comparing to the base model after SFT and GRPO, but without the REPA. I could not find this comparison in the paper. If I missed it, please point me to it and highlight it more prominently in the paper, or provide results in dsicussion.

1 comment
AuthorsOpenReview ↗

Response to Reviewer CPiQ

Thanks for your careful reading and positive feedback on our experiments and presentation! We address your concerns below.

CPiQ-W1

First, this work feels relatively incremental. I feel it is somewhat obvious (given that REPA and similar methods have been around for 2 years now), that if we improve the visual representation within a language model for specific tasks (i.e. 3D), we will get better results on related tasks.

REPA is a good foundation for our work. Its success motivates geometric supervision, but does not tell us how alignment should account for camera motion.

Camera motion changes the view of a scene while its geometry remains unchanged. We use SE(3) to bring camera rotation and translation into the alignment process, so that geometric supervision explicitly accounts for viewpoint changes. The gains over naive alignment across three spatial tasks also support this pose-aware design (original Tabs. 3 and 7).

CPiQ-W2

A core work that authors appear not to have cited or discussed is Yoon et al, 2025. This work introduces REPA for multimodal LLMs (using the original DINOv2) and shows solid increases in perceptual tasks. I feel that it is conceptually similar to the core insight of this paper, and should be adequately discussed before publication.

Thanks for pointing it out! We acknowledge that VIRAL is closely related to our work and should have been discussed.

VIRAL addresses the loss of useful visual information under text supervision. Our work shares this motivation, and we have added the citation and discussion to §2. However, we focus more specifically on spatial tasks, using SE(3) to account for camera motion in alignment. We also discuss the role of spatial understanding in general-purpose MLLMs in Q1.

CPiQ-Q1

Why specialize a general-purpose MLLM for spatial tasks? Does SE(3) alignment improve spatial representations at a significant cost to other visual capabilities, such as chart understanding and pointing?

We agree that general-purpose vision is important. Our focus on spatial understanding is motivated by applications such as embodied navigation and robotic inspection, which require reasoning about locations, distances, and routes. In these settings, smaller models with stronger spatial reasoning can be more useful than larger general-purpose models.

We also evaluate general visual abilities, including reading text in images, visual perception, and multimodal reasoning. Tab. 6 (original) shows some degradation after spatial training, but the declines are modest. Our SFT includes general video data from LLaVA-Hound for retaining visual capabilities.

ModelMMMUTextVQAMMBenchMMVet
Qwen2.5-VL-7B58.684.982.667.1
Geo-REPA-7B57.384.279.864.5

CPiQ-Q2

Authors first do an SFT stage and then a GRPO stage. If we do a FLOP-for-FLOP analysis, how much (if at all) does the GRPO outperform a continued SFT?

Good point. We compare the two training routes on VSI-Bench using Geo-REPA-3B. For a practical comparison, we use training time on the same hardware as the budget. Both start from the same SFT checkpoint and run for 34 additional hours. Continued SFT uses the same QA data as GRPO.

TrainingTimeVSI Avg.
SFT21h50.1
Continued SFT55h53.3
Cold-start + GRPO55h59.4

We choose GRPO to improve reasoning with task QA data that lacks CoT annotations. Continued SFT on these data supervises the answers, whereas GRPO lets the model generate its own reasoning and learn from answer rewards, as in many prior works including Spatial-MLLM.

We have added this comparison to §4.4.

CPiQ-Q3

In my opinion, the fair comparison is not comparing to the base model, but comparing to the base model after SFT and GRPO, but without the REPA. I could not find this comparison in the paper. If I missed it, please point me to it and highlight it more prominently in the paper, or provide results in discussion.

Thanks for pointing this out! The Baseline in Tab. 3 (original) and No alignment in Tab. 7 (original) both follow the same SFT, cold-start, and GRPO pipeline as Geo-REPA, without REPA at any stage. We summarize the 3B results below.

MethodVSI Avg.SQA3D EM-1Scan2Cap CIDEr
No alignment52.651.561.7
Geo-REPA-3B59.456.880.1

Thank you again for helping us explain our motivation and training choices more clearly. We have incorporated the discussion and revisions into the paper.

Reviewer_Vi9jOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary
  1. GEO-REPA transfers geometric priors from a frozen VGGT teacher into intermediate MLLM representations through joint visual- feature and pose alignment.

  2. Its central design predicts per-frame rigid transformations and applies them to projected visual tokens before alignment; the geometry teacher and auxiliary modules are removed at inference.

  3. Experiments report competitive spatial performance across VSI-Bench, SQA3D, and Scan2Cap, with matched alignment ablations and limited cross-backbone validation.

Critical strengths
  1. Practical inference design. Geometry supervision is confined to training, avoiding dependence on an external geometry encoder or auxiliary depth inputs at deployment.

  2. Consistent gains over direct alignment. Matched ablations favor the proposed design across all three tasks—for example, VSI- Bench improves from 56.7 to 59.4 and SQA3D EM-1 from 54.3 to 56.8 (Table 7).

  3. Useful geometric diagnostics. Evaluation against ground-truth camera poses provides stronger evidence than feature visualizations alone: rotation error decreases from 47.8° with shuffled poses to 8.6° (Table 8).

Critical weaknesses
  1. The equivariance claim is insufficiently established. Applying rigid transformations to learned feature triplets does not establish that the projected features or subsequent unconstrained MLP are equivariant. The assumed coordinate-frame interpretation of VGGT embeddings also lacks direct justification (§3.2).

  2. Ablations do not isolate the central contribution. The full method combines pose supervision, additional prediction heads, and structured transformations. Existing comparisons do not establish whether gains specifically require SE(3) structure rather than additional supervision or model capacity.

  3. More methods need to be discussed, like training-free spatial enhancement, See&trek: Training-free spatial prompting for multimodal large language model

Questions for authors
  1. Section 3.2 defines equivariance but does not establish it for the proposed alignment mapping. Equation (3) specifies an SE(3) action on latent feature triplets, whereas the learned projection and subsequent MLP are not shown to preserve equivariance. Can the authors specify the relevant group actions and prove the claimed property, or clarify that the method is pose-conditioned alignment rather than an equivariant mapping? I think it's biggest problem of the paper.

  2. Does SE(3) structure itself drive the gains? Can you compare against pose-supervision-only and parameter-matched unconstrained transformations, while keeping training data and optimization identical?

  3. Can you make the evaluation auditable? Please provide exact GRPO train/evaluation splits, scene-level overlap checks across training stages, and multi-seed results for the principal comparisons.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Vi9j

We thank you for recognizing our efficiency, effectiveness, and pose diagnostics. We address your comments point by point below.

Vi9j-W1

The equivariance claim is insufficiently established. Applying rigid transformations to learned feature triplets does not establish that the projected features or subsequent unconstrained MLP are equivariant. The assumed coordinate-frame interpretation of VGGT embeddings also lacks direct justification (§3.2).

This is an important distinction. Eq. 3 defines an SE(3) action on projected feature triplets, without establishing strict equivariance of the learned alignment mapping. The projection and subsequent MLP are not constrained to preserve equivariance. We therefore describe Geo-REPA as pose-aware alignment with SE(3) structure.

VGGT embeddings serve as geometric supervision targets, without assuming that either feature space has an explicit physical coordinate interpretation. The comparisons in W2 assess the empirical benefit of the SE(3) structure.

Vi9j-W2

Ablations do not isolate the central contribution. The full method combines pose supervision, additional prediction heads, and structured transformations. Existing comparisons do not establish whether gains specifically require SE(3) structure rather than additional supervision or model capacity.

We additionally evaluate two variants on Scan2Cap with Qwen2.5-VL-3B. The unconstrained variant predicts an unrestricted (3\times3) matrix from the same pose tokens and applies it to the projected feature triplets in place of the rotation matrix. It retains the translation branch and hidden layers, with a slightly larger output head.

MethodPose sup.CIDEr
Naive alignment✗72.3
Naive alignment✓75.8
Unconstrained transform✓77.4
Geo-REPA-3B✓80.1

The unconstrained transform uses the same hidden layers as Geo-REPA and a slightly larger output head, yet achieves lower CIDEr. These results support the benefit of SE(3) structure beyond additional pose supervision and parameters.

We have added these comparisons to §4.4.

Vi9j-W3

More methods need to be discussed, like training-free spatial enhancement, See&trek: Training-free spatial prompting for multimodal large language model

Thank you for pointing out this omission. We have expanded §2 to discuss training-free spatial enhancement, including See&Trek. See&Trek shows that supplying camera motion cues through input prompts can improve spatial reasoning without updating the MLLM. Our contribution addresses how to learn from such geometric information through pose-aware alignment of intermediate representations. This distinguishes our focus on representation learning from spatial prompting at inference.

We have added this discussion to §2.

Vi9j-Q1

Eq. (3) defines an SE(3) action, but equivariance of the learned projection and MLP is not established. Please specify the group actions and prove equivariance, or clarify the method as pose-conditioned alignment.

Please refer to our response to W1.

Vi9j-Q2

Does SE(3) structure itself drive the gains? Can you compare against pose-supervision-only and parameter-matched unconstrained transformations, while keeping training data and optimization identical?

Please refer to our response to W2.

Vi9j-Q3

Can you make the evaluation auditable? Please provide exact GRPO train/evaluation splits, scene-level overlap checks across training stages, and multi-seed results for the principal comparisons.

For VSI-Bench evaluation, GRPO uses 100K samples randomly drawn from MM-Spatial, SPAR, and MindCube, without using VSI-Bench QA pairs. For SQA3D and Scan2Cap, GRPO uses the standard training splits, and evaluation uses the validation splits. We checked scene names in the SFT, cold-start, and GRPO data against the evaluation sets and found no matches. We have clarified the training data and overlap checks in §4.1.

We report single-seed results due to the cost of repeating the full training pipeline. Although variability across seeds remains unquantified, our results support two aspects of our contribution: pose-aware alignment improves performance across three tasks (original Tabs. 3 and 7), and SE(3) structure provides further gains beyond additional pose supervision and model capacity, as shown by the ablations in W2.

Thank you again for your constructive feedback. We have incorporated the discussion and revisions into the paper.

Reviewer_DuPSOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Geo-REPA adds training-time geometry supervision to a video MLLM. Pose tokens predict per-frame rigid transformations, which are applied to projected intermediate visual features before the features are aligned with a VGGT geometry teacher; a separate objective aligns the pose tokens with the teacher's pose representations. The teacher and alignment modules are removed at inference. The paper evaluates spatial video reasoning, situated question answering, and object captioning. It compares the full method with no-alignment and direct-alignment variants and includes pose diagnostics and evaluations on general multimodal benchmarks.

Critical strengths
  1. Across the three spatial tasks, the full method outperforms the no-alignment and direct-alignment variants under matched pipelines, which support the combined training approach. The additional Scan2Cap result on Qwen3-VL extends the evidence to another backbone.

  2. The additions are used only during training, so the spatial gains come without additional modules at inference. The ScanNet camera-pose errors show that the predicted transformations capture camera orientation and translation direction. The reported declines on general multimodal benchmarks also document the trade-off of spatial post-training.

Critical weaknesses

The ablations support the full method, but as documented they do not isolate the contribution of the pose-conditioned rigid transformation.

  • Training settings are matched, but the paper does not say whether the direct-alignment baseline keeps the pose tokens and pose-alignment loss, or uses a projection of comparable capacity.
  • Depending on which components differ, some of the gain could come from additional pose supervision or from the alignment branch's capacity rather than from the transformation itself.
  • The pose-error and perturbation analyses show that the predicted transformations carry camera-motion information and affect alignment, but neither measures their effect on task accuracy.
Questions for authors
  1. Specify whether the direct-alignment baseline retains the pose tokens, pose-alignment loss, and projection architecture used by the full method.

  2. Minor clarification: Identify the corpus used for VSI-Bench-specific GRPO.

  3. Minor clarification: The Scan2Cap tables report CIDEr, BLEU-4, METEOR, and ROUGE without an IoU threshold.

1 comment
AuthorsOpenReview ↗

Response to Reviewer DuPS

We appreciate your recognition of our effectiveness and efficiency. We address your comments point by point below.

DuPS-W1

Training settings are matched, but the paper does not say whether the direct-alignment baseline keeps the pose tokens and pose-alignment loss, or uses a projection of comparable capacity.

The original naive alignment removes the pose tokens, pose-alignment loss, and SE(3) transformation, directly aligning visual features with VGGT. It retains the same visual projection architecture as Geo-REPA. This comparison evaluates the combined pose-aware design.

To further control supervision and capacity, the additional comparisons in W2 retain pose tokens and pose supervision, with other alignment components fixed.

We have clarified the baseline configuration in §4.4 of the revised paper.

DuPS-W2

Depending on which components differ, some of the gain could come from additional pose supervision or from the alignment branch's capacity rather than from the transformation itself.

Valuable suggestion for isolating the contribution. We additionally evaluate two variants on Scan2Cap with Qwen2.5-VL-3B. The unconstrained variant predicts an unrestricted (3\times3) matrix from the same pose tokens and applies it to the projected feature triplets in place of the rotation matrix. It retains the translation branch and hidden layers, with a slightly larger output head.

MethodPose sup.CIDEr
Naive alignment✗72.3
Naive alignment✓75.8
Unconstrained transform✓77.4
Geo-REPA-3B✓80.1

These results support the benefit of SE(3) structure beyond additional pose supervision and parameters.

We have added these comparisons to §4.4.

DuPS-W3

The pose-error and perturbation analyses show that the predicted transformations carry camera-motion information and affect alignment, but neither measures their effect on task accuracy.

Tab. 8 and Fig. 5b (original) characterize pose prediction and alignment sensitivity, respectively. The controlled comparisons in W2 evaluate the contribution to downstream performance. With pose tokens and pose supervision retained, Geo-REPA improves Scan2Cap CIDEr over naive alignment and the unconstrained transform, supporting the benefit of the SE(3) structure for task performance.

We have added these comparisons to §4.4.

DuPS-Q1

Specify whether the direct-alignment baseline retains the pose tokens, pose-alignment loss, and projection architecture used by the full method.

Please refer to our response to W1.

DuPS-Q2

Minor clarification: Identify the corpus used for VSI-Bench-specific GRPO.

Thank you for pointing this out. For VSI-Bench evaluation, our GRPO training uses 100K samples randomly drawn from MM-Spatial, SPAR, and MindCube, consisting of images, questions, and ground-truth answers. No VSI-Bench QA pairs are used for GRPO.

We have clarified the training data in §4.1.

DuPS-Q3

Minor clarification: The Scan2Cap tables report CIDEr, BLEU-4, METEOR, and ROUGE without an IoU threshold.

Following VG-LLM, we report all Scan2Cap metrics at a 3D box IoU threshold of 0.5. We have clarified this in §4.1.

Thank you again for your constructive feedback. We have incorporated the discussion and revisions into the paper.

Reviewer_HmdfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Geo-REPA, a geometry-aware representation alignment framework designed to improve the spatial reasoning capabilities of multimodal large language models. The authors argue that existing input-level geometry fusion methods do not explicitly encourage geometric information to be preserved in intermediate representations.

To address this issue, Geo-REPA introduces pose-aware representation alignment. It predicts camera rotations and translations from learnable pose tokens, uses the predicted transformations to align intermediate visual features with VGGT representations, and adds pose-level supervision during training. The auxiliary geometry modules are removed at inference.

The method is evaluated on VSI-Bench, SQA3D, and Scan2Cap, with additional experiments on representation alignment, camera pose prediction, training stages, and backbone transfer. The reported results show improvements over direct feature alignment and competitive performance against existing spatial reasoning methods.

Critical strengths
  1. The method shows consistent gains over direct representation alignment. The matched experiments in Tables 3 and 7 are a strength of the paper. The proposed alignment outperforms both no alignment and naive alignment across three different tasks. In particular, the improvements on VSI-Bench and SQA3D suggest that the additional pose-aware design is useful beyond dense captioning. These controlled comparisons provide stronger evidence than simply comparing the final model with published results from other methods.

  2. The training-time supervision is practical, and the results are not limited to a single backbone. The geometry encoder and auxiliary alignment modules are not required at inference, avoiding the cost of running an additional geometry model during deployment. The improvement on Qwen3-VL-8B for Scan2Cap is also encouraging. The representation and pose analyses provide useful supporting evidence, although I have some reservations about how these results are interpreted.

Critical weaknesses
  1. The novelty over existing 3D representation alignment methods is not sufficiently established. The motivation of Geo-REPA is closely related to 3DRS, which also improves MLLM spatial understanding through representation supervision from pretrained 3D foundation models. Although Geo-REPA introduces pose-aware transformations, it remains unclear whether this design offers a substantial advantage over existing 3D-aware alignment approaches. A comparison under the same backbone and training pipeline would help clarify the contribution.

  2. The ablation studies do not sufficiently isolate the effects of different components. Tables 3 and 7 show consistent improvements over naive alignment, but pose supervision, learnable pose tokens, and SE(3) transformations are introduced together. Therefore, the improvements cannot be attributed to the geometric transformation alone. Additional comparisons that retain pose supervision without SE(3) transformation, or replace it with a parameter-matched learned transformation, would make the results more convincing.

  3. The experimental analysis does not fully validate the motivation for pose-aware alignment. A central motivation of Geo-REPA is to address coordinate-frame differences caused by camera motion. However, the experiments do not directly establish whether its advantage over naive alignment becomes more significant under larger viewpoint changes or more complex camera trajectories. The pose perturbation analysis and visualizations provide supporting evidence, but they do not directly measure downstream performance under different motion conditions. A more targeted evaluation separating low-motion and high-motion sequences would better support the proposed design.

  4. The training data and the contribution of different training stages require further clarification. The paper combines geometry-oriented SFT data, generated reasoning trajectories, and benchmark-specific GRPO. While the matched alignment ablations control for the training pipeline, the description of the task-specific RL data remains incomplete. In particular, the source of the VSI-Bench training data should be clarified, since the official benchmark is released as an evaluation set. Table 9 reports the effects of training stages on Scan2Cap, but a corresponding analysis on VSI-Bench and SQA3D would help determine whether the benefits of geometry-aware alignment are consistent before and after task-specific RL.

I currently lean toward weak rejection due to the concerns discussed above. However, I would be happy to reconsider and raise my rating if the authors can provide convincing clarifications or additional experimental evidence addressing these concerns.

Questions for authors

Q1. How does Geo-REPA compare with 3DRS under matched training and backbone settings? What benefit can be specifically attributed to the pose-aware alignment design?

Q2. Can the authors separate the contributions of pose supervision, SE(3) transformation, and additional learnable parameters through more targeted ablations?

Q3. Does Geo-REPA provide larger improvements over naive alignment on videos with substantial camera motion or large viewpoint changes? Could the authors report performance grouped by camera motion magnitude?

Q4. What training data are used for VSI-Bench-specific GRPO, and how are they separated from the evaluation scenes? Are the alignment improvements consistent before and after RL on all three benchmarks?

Q5. Does the method transfer to newer architectures, such as Qwen3.5-9B, beyond the reported Qwen3-VL-8B Scan2Cap experiment? How sensitive is the alignment strategy to changes in backbone architecture and feature representations?

2 comments
AuthorsOpenReview ↗

Response to Reviewer Hmdf (1/2)

We thank you for recognizing our controlled ablations, inference efficiency, and backbone transfer. We address your comments point by point below.

Hmdf-W1

The novelty over 3DRS, which also uses pretrained 3D models for representation supervision, remains unclear. The added benefit of pose-aware transformations should be evaluated under the same backbone and training pipeline.

Both 3DRS and Geo-REPA use pretrained 3D representations as supervision. Our contribution is to make the alignment explicitly pose-aware. Geo-REPA predicts per-frame rotations and translations from learned pose tokens, transforms intermediate visual features before matching geometry features, and jointly supervises pose representations.

Empirically, we summarize the existing results from Tabs. 1–4 and 7 (original) below. The reported method comparison uses the respective backbones and training pipelines. The matched ablation uses Qwen2.5-VL-3B with identical training data, frame sampling, and SFT, cold-start, and GRPO settings.

MethodVSI-Bench Avg.SQA3D EM-1Scan2Cap CIDEr
3DRS-7B45.960.686.1
Geo-REPA-7B65.463.690.2
Naive alignment (3B)56.754.372.3
Geo-REPA-3B59.456.880.1

Geo-REPA achieves stronger reported results than 3DRS, while the ablations demonstrate consistent benefits of the combined pose-aware design over direct alignment across all three tasks.

We have clarified the distinction from 3DRS in §2 and the scope of ablations in §4.4 of the revised paper.

Hmdf-W2

Although Tables 3 and 7 show gains over naive alignment, pose supervision, learnable pose tokens, and SE(3) transformations are introduced together. Please isolate the transformation’s contribution through comparisons with pose supervision alone and a parameter-matched learned transformation.

Valuable suggestion! We additionally conduct two ablations on Scan2Cap with Qwen2.5-VL-3B. The unconstrained variant predicts an unrestricted (3\times3) matrix from the same pose tokens and applies it to the projected feature triplets in place of the rotation matrix. It retains the translation branch and hidden layers, with a slightly larger output head.

MethodPose sup.CIDEr
Naive alignment✗72.3
Naive alignment✓75.8
Unconstrained transform✓77.4
Geo-REPA-3B✓80.1

Pose supervision improves naive alignment. Among the pose-supervised variants, Geo-REPA outperforms both alternatives, supporting the benefit of the SE(3) structure beyond additional pose supervision and transformation capacity.

We have added these comparisons to §4.4.

Hmdf-W3

The current analyses do not show whether gains over naive alignment increase with viewpoint changes or camera motion. Pose perturbations and visualizations provide indirect evidence. Please compare downstream performance on low- and high-motion sequences.

Good suggestion for testing the motivation. We split SQA3D scenes at the median of their mean camera rotation relative to the first input frame and compare the 7B models using EM-1.

MotionNaiveGeo-REPAΔ
Low61.263.9+2.7
High59.863.4+3.6

Geo-REPA outperforms naive alignment in both groups. Although both models score lower in the high-motion group, Geo-REPA shows a smaller decrease and a larger advantage. This pattern is consistent with our motivation that accounting for camera pose during alignment helps handle larger differences across input views.

AuthorsOpenReview ↗

Response to Reviewer Hmdf (2/2)

Hmdf-W4

Please clarify the task-specific GRPO data, particularly for VSI-Bench, whose official release is an evaluation set. Beyond the Scan2Cap analysis in Table 9, please examine whether alignment gains remain consistent before and after RL on VSI-Bench and SQA3D.

For VSI-Bench evaluation, our GRPO training uses 100K samples randomly drawn from MM-Spatial, SPAR, and MindCube, consisting of images, questions, and ground-truth answers. No VSI-Bench QA pairs are used for GRPO.

We additionally compare the Qwen2.5-VL-3B variants before and after GRPO on VSI-Bench and SQA3D. “Before” denotes the checkpoint after cold start.

MethodStageVSI Avg.SQA3D EM-1
Naive alignmentBefore51.049.5
Naive alignmentAfter56.754.3
Geo-REPABefore54.352.7
Geo-REPAAfter59.456.8

Both methods improve with RL, while Geo-REPA outperforms naive alignment at both stages on both benchmarks. Its advantage is already present before RL and persists afterward. These results complement the existing Scan2Cap training stage analysis in Tab. 9 (original).

Hmdf-Q1

How does Geo-REPA compare with 3DRS under matched training and backbone settings? What benefit can be specifically attributed to the pose-aware alignment design?

Please refer to our responses to W1 and W2.

Hmdf-Q2

Can the authors separate the contributions of pose supervision, SE(3) transformation, and additional learnable parameters through more targeted ablations?

Please refer to our response to W2.

Hmdf-Q3

Does Geo-REPA provide larger improvements over naive alignment on videos with substantial camera motion or large viewpoint changes? Could the authors report performance grouped by camera motion magnitude?

Please refer to our response to W3.

Hmdf-Q4

What training data are used for VSI-Bench-specific GRPO, and how are they separated from the evaluation scenes? Are the alignment improvements consistent before and after RL on all three benchmarks?

We compared scene names between the GRPO training data and VSI-Bench and found no matches. Please refer to our response to W4 for the training data and stage comparisons.

Hmdf-Q5

Does the method transfer to newer architectures, such as Qwen3.5-9B, beyond the reported Qwen3-VL-8B Scan2Cap experiment? How sensitive is the alignment strategy to changes in backbone architecture and feature representations?

Thank you for suggesting this extension. Our Qwen3-VL-8B experiment improves Scan2Cap CIDEr from 79.7 to 94.1 (Tab. 5), while retaining the same geometry teacher, alignment objectives, and training strategy. We have not yet evaluated Qwen3.5-9B.

Learned projections accommodate different feature dimensions, while the alignment layer may need adjustment for different backbones. Fig. 6b shows that neighboring alignment layers remain competitive on Qwen2.5-VL. This provides evidence about layer selection within the tested backbone family, while sensitivity to broader architectural changes remains untested. We have clarified this in §4.3.

Thank you again for your constructive feedback. We have incorporated the discussion and revisions into the paper.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Responses to all Reviewers

Dear Chairs and Reviewers,

Thank you for your constructive and detailed feedback.
This note addresses three concerns shared across the reviews.

1. Contribution beyond Existing Alignment Methods (Hmdf, Vi9j, CPiQ)
Camera motion changes the view of a scene while its geometry remains unchanged. Geo-REPA uses SE(3) to bring camera rotation and translation into representation alignment, so that geometric supervision accounts for these viewpoint changes. We clarified this motivation and formulation in §1 and §3.2 of the revised paper, and expanded the discussion of 3DRS, VIRAL, and See&Trek in §2.

2. Contribution of SE(3) Structure (Hmdf, DuPS, Vi9j, CPiQ)
Comparisons with no-alignment baselines use the full training pipeline and show gains from adding alignment. The additional Scan2Cap ablations further show that Geo-REPA outperforms naive alignment with pose supervision and an unconstrained transformation, supporting the benefit of SE(3) structure beyond additional supervision and capacity. We clarified the baselines and added these comparisons to §4.4.

3. Role of Reinforcement Learning (Hmdf, CPiQ)
The gains over naive alignment are present on VSI-Bench and SQA3D both before and after GRPO. For task QA data without CoT annotations, we use GRPO to learn reasoning from answer rewards. The added comparison shows that cold-start followed by GRPO outperforms continued SFT with equal additional training time on the same hardware, supporting this training choice. We added this comparison and expanded the training stage analysis in §4.4.

We hope these clarifications address the shared concerns and help explain our design and results.
Thank you again for your time and thoughtful comments.

Sincerely,
The Authors

AuthorsOpenReview ↗

Summary of Revisions to the Paper

Dear Chairs and Reviewers,

We appreciate your constructive feedback and the opportunity to revise our paper.
This document summarizes the main changes made in the revised version.
Separate responses will address shared concerns and individual comments.

Below we list the key revisions:

  1. We refined the abstract to describe pose-aware alignment with SE(3) structure more clearly.
  2. We clarified the camera motion motivation and Geo-REPA’s contribution in §1.
  3. We expanded the discussion of 3DRS, VIRAL, and See&Trek in §2.
  4. We clarified the SE(3) action and VGGT supervision in §3.2.
  5. We specified the GRPO data sources and dataset splits in §4.1.
  6. We reported scene-name overlap checks between training and evaluation data in §4.1.
  7. We specified the IoU threshold of 0.5 for Scan2Cap metrics in §4.1.
  8. We specified the use of a single training seed in §4.1.
  9. We clarified backbone transfer and alignment layer selection in §4.3.
  10. We clarified the training pipeline and components of the alignment baselines in §4.4.
  11. We added ablations on pose supervision and SE(3) structure in Tab. 3 and §4.4.
  12. We discussed alignment gains before and after GRPO on VSI-Bench and SQA3D in §4.4.
  13. We compared continued SFT with cold-start + GRPO under the same time budget in §4.4.

We hope these revisions help clarify our method and address the main concerns raised in the reviews.
Thank you again for your time and thoughtful comments.

Sincerely,

The Authors

AuthorsOpenReview ↗

Revisions Following Your Feedback

Thanks for carefully reading our revised paper and suggesting these improvements

We revise as follows:

  1. We rewrote §1 to clarify how Geo-REPA builds on REPA. We explain how weaker geometry alignment in later layers of input-level fusion models motivates intermediate supervision to help the backbone preserve and use 3D cues beyond the input stage.
  2. We also clarified the contribution of pose-aware alignment. Camera motion changes the view while scene geometry stays the same. We explain how SE(3) brings rotation and translation into geometric supervision, and connect this design to the analyses of camera motion and spatial performance.
  3. We revised the abstract and conclusion to state these contributions directly and summarize the experimental findings that support them.
  4. In §4.4, we connected the comparisons more directly to our design choices. The discussion explains the gains from geometry supervision under the same training pipeline, the added benefit of SE(3) structure, and the advantage of cold start + GRPO over continued SFT under the same time budget.

Thanks again for helping us make the paper clearer!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.