acceptodds
Under review as a conference paper at ICLR 2027

Feeling the Space: Egomotion-Aware Video Representation for Efficient and Accurate 3D Scene Understanding

Abstract

Recent Multimodal Large Language Models (MLLMs) have shown high potential for spatial reasoning within 3D scenes. However, they typically rely on computationally expensive 3D representations like point clouds or reconstructed Bird's-Eye View (BEV) maps, or lack physical grounding to resolve ambiguities in scale and size. This paper significantly enhances MLLMs with egomotion modality data, captured by Inertial Measurement Units (IMUs) concurrently with the video. In particular, we propose a novel framework, called Motion-MLLM, introducing two key components: (1) a cascaded motion-visual keyframe filtering module that leverages both IMU data and visual features to efficiently select a sparse yet representative set of keyframes, and (2) an asymmetric cross-modal fusion module where motion tokens serve as intermediaries that channel egomotion cues and cross-frame visual context into the visual representation. By grounding visual content in physical egomotion trajectories, Motion-MLLM can reason about absolute scale and spatial relationships across the scene. Our extensive evaluation shows that Motion-MLLM makes significant improvements in various tasks related to 3D scene understanding and spatial reasoning. Compared to state-of-the-art (SOTA) methods based on video frames and explicit 3D data, Motion-MLLM achieves competitive accuracy while running and faster, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal large language models, 3D scene understanding, spatial reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TocbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • Motion-MLLM adds IMU motion data to a VLM to improve spatial understanding.

  • It uses motion-guided keyframe selection and asymmetric cross-attention to combine visual and motion features.

  • Evaluation covers five benchmarks with synthetic IMU and 120 questions over six real-IMU sequences, reporting improved accuracy and lower latency.

Critical strengths
  • Meaningful gains: VSI-Bench improves from 49.0 to 58.2 over the matched-protocol VGGT-only baseline, particularly on distance and room-size estimation.

  • Useful ablations: Encoder comparisons, motion corruption, scale interventions, and sensor perturbations provide evidence that the model uses aligned motion information.

Critical weaknesses
  • Limited real-data validation: A.4 does not explain how camera trajectories yield object-distance and room-size ground truth. The six TUM-VI room sequences also offer limited environmental diversity, weakening the evidence for broad real-world transfer.

  • Unclear system dependencies and costs: VGGT feature reuse and latency accounting are insufficiently specified. ScanRefer [email protected] rises from 19.6 to 55.3 with proposal refinement, so the role of external 3D proposals must be distinguished from the model’s own grounding ability.

Questions for authors
  • How are TUM-VI distance and room-size answers measured and verified? Do all six sequences depict the same room?

  • Which frames are jointly processed by VGGT, and are features recomputed or cached? What operations and hardware are included in latency measurements?

  • What external 3D information generates the ScanRefer proposals and Scan2Cap coordinates? Where are reference camera poses used?

  • How is the motion gate initialized? Does Table 13 actually block cross-frame relay, given that each updated motion token can already contain other frames’ visual information?

4 comments
AuthorsOpenReview ↗

Kindly requesting for feedback on our rebuttal

Dear Reviewer,

Thank you again for your detailed review. We have posted a response addressing each question: the TUM-VI ground-truth procedure and the shared room, VGGT feature reuse and latency accounting (with a component breakdown in Tab. 6), the role of external proposals in ScanRefer and Scan2Cap, the motion gate initialization, and the scope of the relay intervention (Tab. 14). A revised paper has been uploaded with all changes highlighted in red.

We would be grateful to know whether these address your concerns, in particular the real-data validation and the system dependencies. If anything remains unclear, we would welcome further discussion.

As there are only a few days remaining in the discussion period, we would appreciate any additional comments at your earliest convenience.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Kind reminder before the discussion period closes

Dear Reviewer,

As the discussion deadline approaches, we would be very grateful for any response to our rebuttal. We have responeded each of your points and submitted a revised paper version with changes highlighted, including several new experiments conducted for this response, and we remain ready to clarify anything that is still unclear.

Even a brief note on whether our rebuttal has resolved your concerns would be valuable to us, and we would gladly answer any remaining questions in the time available.

Thank you very much for your time and consideration.

AuthorsOpenReview ↗

Gentle follow-up as the discussion period closes

Dear Reviewer,

Thank you again for your review. We wanted to follow up as the discussion period closes tomorrow. We have posted a detailed response addressing each of your questions, including the TUM-VI ground truth, VGGT feature reuse, latency accounting, the role of external proposals, and the relay intervention. A revised paper with all changes highlighted is also available.

We would be grateful for any feedback at your convenience.

Thank you for your time.

AuthorsOpenReview ↗

Final follow-up before the discussion period closes

Dear Reviewer,

As the discussion period ends today, we wanted to reach out one last time. We would greatly appreciate any feedback on whether our response and revised paper address your concerns. We are happy to clarify anything further.

Thank you for your time.

Reviewer_jo7uOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adds egomotion from an IMU as an input to video MLLMs for 3D scene understanding. The motivation is that video-only models can't recover metric scale, while 3D-input models need depth or point clouds. The proposed Motion-MLLM is built on Qwen2.5-VL-3B with a VGGT geometry encoder, following Spatial-MLLM. It adds two components:

  1. Cascaded motion-visual keyframe filtering. To select a small set of informative, non-redundant keyframes, each frame passes through three gates: an IMU-based motion gate, a sparse-feature parallax check, and a visual-token cosine-distance check. Only frames that survive the first two are encoded by the 2D and VGGT encoders.

  2. Asymmetric cross-modal fusion. A bidirectional GRU encodes the IMU segment between each pair of keyframes into one motion token. Visual and motion tokens then attend to each other in both directions, followed by a layer where only visual tokens attend to motion tokens. This lets motion tokens relay context between frames. Only the enriched visual tokens go to the LLM.

The existing benchmarks have no IMU data, so the authors synthesize it from each dataset's ground-truth camera poses using B-spline differentiation and a MEMS noise model. They also build a small 120-question benchmark on TUM-VI with real IMU recordings.

Critical strengths
  • The core idea is well motivated and practical. IMUs are cheap and present on phones, robots, and autonomous vehicles. Using measured metric egomotion to resolve scale ambiguity in video-only spatial reasoning is a sensible direction. According to Appendix B.3-B.4, the support for the utilization of IMU is provided by:

    • Feeding zeros brings the model back to the no-IMU baseline.
    • Shuffled or mismatched IMU streams perform below that baseline.
  • Consistent gains from egomotion across tasks and datasets. Compared with the VGGT-only baseline (Spatial-MLLM's configuration retrained under the same pipeline), adding IMU raises the VSI-Bench average from 49.0 to 58.2 (Table 6). The largest gains are on Abs.DistAbs. Dist +16.5, RoomSizeRoom Size +17.3 and Rel.DistRel. Dist +16.0 (Table 17).

  • A real-sensor check. The same checkpoint on real IMU and mocap-synthesized IMU gives nearly identical results.

Critical weaknesses
  1. The efficiency claim conflates several sources of speedup. Motion-MLLM (4B) matches Video-3D LLM (7B, depth + pose input) on ScanQA at every frame budget: 29.8 vs 29.5 EMEM with adaptive selection, 31.5 vs 31.7 at 128 frames (Table 5). However, the reported 1.30× and 1.61× speedups come from different sources, which the paper attributes collectively to egomotion-aware filtering:
  • Against Video-3D LLM (1.61×), the speedup mainly reflects the smaller LLM (4B vs 7B). At equal frame counts, Motion-MLLM is no faster than Spatial-MLLM, which has the same size: 0.90 vs 0.88 s at 32 frames, and 3.10 vs 3.06 s at 128 frames.
  • Against Spatial-MLLM (1.30×), it is a comparison of frame selectors (MV filtering vs max-coverage sampling), not of models. The IMU gate does make selection cheaper: the cascade lets the encoders skip most frames (0.61 s vs 0.92 s for visual-only sampling, Table 6). But most of the frame reduction comes from the vision-based parallax check (~80 $\rightarrow ~30 frames, Table 7). No ablation runs the cascade without the IMU gate.
  • Accuracy gain from MV filtering is minimal. It adds only +0.7 EMEM over uniform 32-frame sampling. Stage 1 alone (~80 frames) is more accurate than the full cascade (30.5 vs 29.8 EMEM; 58.8 vs 58.2 VSIVSI), so the later stages trade accuracy for speed.

Disregarding the latency of IMU synthesis (i.e., assuming real IMU input is available), the introduction's claim that Motion-MLLM is "1.30× and 1.61× faster … at matched accuracy, demonstrating that egomotion grounding delivers stronger spatial reasoning … while avoiding the overhead of explicit 3D representations" still overstates the evidence.

  • First, accuracy matches only the 3D baseline (29.8 vs 29.5 EMEM against Video-3D LLM), while Motion-MLLM is +3.9 EMEM better than the 2D baseline, Spatial-MLLM.
  • Second, the speedups stem mainly from the smaller LLM (4B vs 7B) and the IMU-gated frame selector, not from egomotion grounding itself. At equal frame counts, Motion-MLLM is no faster than Spatial-MLLM (0.90 vs 0.88 s).
  • Third, the claimed avoidance of "the overhead of explicit 3D representations" is misleading. Video-3D LLM's 3D (depth) input is processed by a parameter-free back-projection and sinusoidal position encoding, so its 3D overhead is negligible, whereas Motion-MLLM runs VGGT, a 1B-parameter geometry transformer, on every keyframe.

A fairer statement would be: Motion-MLLM outperforms a same-size 2D-input MLLM and matches a 7B 3D-input MLLM with a 4B model and without requiring depth sensing, with its lower latency coming mainly from the smaller LLM and a cheaper frame selector.

  1. The novelty claims are overstated. The claim to be "first to use concurrent camera egomotion for scene-level spatial reasoning in MLLMs" needs qualifying. 3D MLLMs (Video-3D LLM, LLaVA-3D) already take ground-truth camera poses, from which Motion-MLLM's IMU is synthesized. More directly, VLM-3R [1] already feeds a video MLLM per-frame view tokens encoding ego-motion, derived from CUT3R [2], which recovers metric-scale geometry and camera pose from monocular video. This challenges both the "first" claim and the motivation that video-only methods cannot recover metric scale. VLM-3R also scores slightly higher than Motion-MLLM on VSI-Bench (58.8 vs 58.2), ScanQA (101.9 vs 100.4 CIDErCIDEr) and SQA3D (60.7 vs 60.2 EM), although it is larger (7B) and trained with VSI-style data. Likewise, 3DRS [3] uses VGGT only as a training-time teacher, yet surpasses Motion-MLLM on ScanQA (104.8 vs 100.4 CIDErCIDEr) and SQA3D (60.6 vs 60.2) without running VGGT at inference. The architecture is also a recombination, as Fig. 1 itself shows. The visual pathway is the 2D-input design of Fig. 1(b) (2D + geometric encoder, as in Spatial-MLLM), and the motion encoder takes the slot of the 3D encoder in Fig. 1(a). The genuinely new parts are therefore the sensor form (raw inertial readings instead of poses) and its use in frame selection.

[1] Fan, Zhiwen, Jian Zhang, Renjie Li, Junge Zhang, Runjin Chen, Hezhen Hu, Kevin Wang et al. "Vlm-3r: Vision-language models augmented with instruction-aligned 3d reconstruction." In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 31054-31065. 2026.

[2] Wang, Qianqian, Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. "Continuous 3d perception model with persistent state." In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10510-10522. IEEE, 2025.

[3] Huang, Xiaohu, Jingjing Wu, Qunyi Xie, and Kai Han. "3drs: Mllms need 3d-aware representation supervision for scene understanding." Advances in Neural Information Processing Systems 38 (2025): 67961-67988.

Questions for authors
  1. Please report Spatial-MLLM and Video-3D LLM with MV filtering on ScanQA/SQA3D, as done for TUM-VI; compare at a matched LLM scale; and break down end-to-end latency into frame selection, visual encoding (2D + VGGT), motion encoding, fusion and LLM decoding for all three models. This would show whether the speedup comes from the selector, model size, or egomotion. The introduction's efficiency claim should then be revised accordingly. The advantage over Video-3D LLM is not requiring depth sensing, rather than lower 3D computation.

  2. Please revise the "first to use concurrent camera egomotion" and SOTA claims. Does IMU add information beyond what a video-based geometry model already recovers?

4 comments
AuthorsOpenReview ↗

Kindly requesting for feedback on our rebuttal

Dear Reviewer,

Thank you again for your detailed review. We have posted a response addressing each point, including new experiments: a latency breakdown with both baselines evaluated on our keyframes (Tab. 6), the cascade run without the motion gate (Tab. 8), and a revised efficiency claim that attributes the speedups to keyframe selection and model size. We have also added VLM-3R and 3DRS to our result tables and revised the SOTA and novelty claims accordingly. A revised paper has been uploaded with all changes highlighted in red.

We would be grateful to know whether these address your concerns, in particular the attribution of the speedups and the positioning relative to VLM-3R and 3DRS. If anything remains unclear, we would welcome further discussion.

As there are only a few days remaining in the discussion period, we would appreciate any additional comments at your earliest convenience.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Kind reminder before the discussion period closes

Dear Reviewer,

As the discussion deadline approaches, we would be very grateful for any response to our rebuttal. We have responeded each of your points and submitted a revised paper version with changes highlighted, including several new experiments conducted for this response, and we remain ready to clarify anything that is still unclear.

Even a brief note on whether our rebuttal has resolved your concerns would be valuable to us, and we would gladly answer any remaining questions in the time available.

Thank you very much for your time and consideration.

Dear Authors,

Thank you for the thorough rebuttal and the additional experiments, which address my main concerns about the efficiency and novelty claims. I will increase my score accordingly.

One issue remains to be fixed in the revision. The latencies in the new Tab. 8 appear inconsistent with those in the original Table 6. The new table reports 9.34 s for Stage 3 alone and 2.92 s for Stages 2+3 without the gate. The original Table 6, however, reported 0.96 s for "Full MV Filtering", which evaluates all criteria on every frame, and 0.92 s for visual-based sampling. Encoding every frame cannot take 0.96 s in one table and 9.34 s in the other. Please reconcile these numbers, state the measurement protocol used for each, and make sure the revised paper reports consistent values, since the claimed 4.8× benefit of the motion gate depends on them.

AuthorsOpenReview ↗

Thanks for reconsideration and response to the follow-up issue

Dear Reviewer,

Thank you for your continued engagement and the time you have invested in this review. We appreciate your recognition of our additional experiments and revised claims. Thank you also for pointing out this inconsistency, the numbers are correct but the two tables operate on different candidate pools, which we should have stated more clearly.

The Frame Sampling Strategy rows in Tab. 7 compare against the sampling strategies used by existing methods such as Spatial-MLLM and Video-3D LLM, which typically pre-sample the video to 3 FPS before frame selection. "Evaluates all criteria on every frame" in the Tab. 7 caption therefore means every pre-sampled frame (~100), so VGGT and the 2D encoder run on ~100 frames (0.96s).

Since our method takes the continuous video and concurrent high-frequency egomotion sensor data as input, Tab. 8 ablates our cascaded filter on the full 30 FPS video (~1,000 frames). "Stage 3 only" encodes all ~1,000 frames (9.34s), and "Stages 2+3" first applies the parallax check to reduce them to ~24 (2.92s). The 4.8× speedup from the motion gate is measured within this full-video setting: it reduces the cascade from 2.92s to 0.61s by discarding ~920 frames before any image is processed.

The 10× difference in candidate frames (~1,000 vs. ~100) accounts for the ~10× difference in latency (9.34s vs. 0.96s). We have submitted a new revised paper and clarified the candidate pool in the captions of Tabs. 7 and 8 and in Sec. 4.5.

Reviewer_FtZfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper proposes Motion-MLLM, which adds synchronized IMU measurements to a video MLLM for 3D scene understanding. The model was initialized from Qwen2.5-VL and VGGT, encodes the IMU segments between keyframes with a GRU, and combines motion and visual features through a two-layer cross-attention module. A separate cascaded filter uses motion, sparse feature tracking, and visual features to select fewer frames. The main argument is that inertial measurements provide metric and cross-view information that is difficult to recover from RGB alone.

  • The quantitative experiments include spatial question answering, visual grounding, and dense captioning. Most use IMU streams synthesized from ground-truth camera trajectories, with an additional evaluation on real TUM-VI recordings.

Critical strengths
  • Personally, I think the main strength of this paper is the usage of IMU measurements for spatial reasoning. This is a useful direction for devices that already capture video and inertial data together (which is very common in all customer devices like smartphones). In addition to recognizing motion or activities, the model also tries to use the signal to understand the surrounding scene.

  • The controlled results are also promising. In Table 6, adding the motion pathway improves VSI-Bench from 49.0 to 58.2 over the VGGT-only baseline. I also liked the analysis in the appendix: the zero/shuffled-IMU controls, scale intervention, and task-level breakdown make a better case that the model uses the motion information than the main comparisons alone. The gains are particularly clear for distance and room-size numerical questions.

  • The paper also includes many useful engineering ablations. Table 7 shows the accuracy/runtime trade-off of the filtering stages, and Table 12 compares several motion encoders.

Critical weaknesses
  • I am also not particularly convinced by the argument that explicitly recovering 3D and feeding them into the MLLM is necessarily too expensive. In a way, Motion-MLLM itself still runs the VGGT backbone, so the additional cost of predicting geometry from the existing features should be lightweight. In Table 5, Motion-MLLM and Spatial-MLLM take 0.90 and 0.88 s with the same 32-frame budget; the larger speed gain appears after filtering to about 21 frames. A useful control would be to give the RGB baseline the same selected frames and measure the additional cost of VGGT camera/depth prediction. Therefore, in my opinion, this could raise the question of: is it worth it to try to train a IMU/motion-specific encoder and use their features for MLLM, or is it better to just infer motion + geometry using off-the-shelf models like VGGT (or anything lighter than VGGT)?
Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Kindly requesting for feedback on our rebuttal

Dear Reviewer,

Thank you again for your thoughtful review. We have posted a response addressing your question, including the control you suggested: Spatial-MLLM evaluated on our keyframes with a full latency breakdown (Tab. 6), and the measured cost of VGGT's depth and camera heads (0.26s vs. 0.02s for our motion encoder). We also discuss why the IMU adds information that VGGT infers poorly on sparse keyframes. A revised paper has been uploaded with all changes highlighted in red.

We would be grateful to know whether these address your concern. If anything remains unclear, we would welcome further discussion.

As there are only a few days remaining in the discussion period, we would appreciate any additional comments at your earliest convenience.

Thank you for your time and consideration.

AuthorsOpenReview ↗

Kind reminder before the discussion period closes

Dear Reviewer,

As the discussion deadline approaches, we would be very grateful for any response to our rebuttal. We have responeded each of your points and submitted a revised paper version with changes highlighted, including several new experiments conducted for this response, and we remain ready to clarify anything that is still unclear.

Even a brief note on whether our rebuttal has resolved your concerns would be valuable to us, and we would gladly answer any remaining questions in the time available.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.