acceptodds
Under review as a conference paper at ICLR 2027

LMM-Track4D: Eliciting 4D Dynamic Reasoning in LMMs via Trajectory-Grounded Dialogue

Abstract

Large multimodal models (LMMs) describe dynamic scenes, but grounding successive questions in an entity’s changing 3D position requires a representation that connects dialogue references to physical time. We formulate trajectory-grounded multi-turn spatiotemporal dialogue, where a model answers questions and returns metric positions over the requested interval. We introduce Track4D-Bench to associate these dialogue references with calibrated observations and timestamped trajectories across 526 clips, 23.5k frames, and 7.5k entity tracks. To retain the queried entity throughout a conversation, we propose LMM-Track4D with a geometry-conditioned target state. This state preserves entity context across turns and selects spatial and temporal evidence for recovering the entity’s trajectory. The resulting representation separates dialogue order from the physical times being queried, allowing later questions to revisit earlier intervals. On Track4D-Bench, LMM-Track4D improves CIDEr by 0.247 over the strongest same-data supervised fine-tuning baseline, alongside gains of 10.2 and 10.5 points in BLEU-4 and METEOR. Geometric evaluation shows a 19.4 percentage-point gain in trajectory-step accuracy over direct coordinate regression. External evaluations further assess dynamic-scene question answering and language-conditioned grounding. These results support geometry-conditioned target states for connecting persistent entity references with metric trajectory reasoning in LMMs. Our code and dataset will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D dynamic reasoning, large multimodal models, multi-turn spatiotemporal dialogue

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ogEjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces LMM-Track4D, a framework for trajectory-grounded multi-turn spatiotemporal dialogue. The authors construct Track4D-Bench, containing 526 video clips with calibrated geometric observations and timestamped entity trajectories. The proposed method combines Ray–Time Geometry Encoding (RTGE), a persistent target-state memory (TRK), and an Object-Slot Kinematic Residual-Anchor (OSK-RA) decoder to support entity-aware dialogue and 3D trajectory prediction. The framework is trained using joint language and geometric supervision followed by GRPO

Critical strengths
  • The paper addresses trajectory-grounded multi-turn dialogue, connecting persistent entity references with temporal and geometric reasoning.
  • The combination of geometric encoding, target-state memory, and trajectory decoding is clearly motivated and reasonably presented.
  • Track4D-Bench provides paired dialogue and geometric supervision, enabling evaluation of both language generation and metric trajectory prediction.
Critical weaknesses
  • The proposed framework primarily integrates existing techniques, including geometry-aware visual encoding, persistent object representations, attention-based evidence selection, and residual trajectory regression. Related approaches such as LLaVA-4D, Ground3D-LMM, and ObjectStream have already explored geometric reasoning, multi-turn spatial grounding, or persistent object representations. Although LMM-Track4D combines these components for trajectory-grounded dialogue, the paper does not sufficiently demonstrate a fundamentally new modeling mechanism or explain why the proposed integration provides a substantial advancement over existing methods.
  • Track4D-Bench contains only 526 clips, with a substantial majority originating from KITTI. The dialogues are constructed through model-assisted generation and filtering, followed by author spot-checks.
  • The main results compare LMM-Track4D, which incorporates geometry-oriented supervision, joint optimization, and GRPO, against baselines primarily trained with language SFT.
  • The controlled ablations in Table 3 show that adding TRK to RTGE improves SAcc and Traj-Acc by only 0.55 percentage points each, and TAcc by 0.72 points. These modest gains are difficult to reconcile with the central emphasis on persistent cross-turn entity memory. Although the standard-objective ablation shows a larger CIDEr improvement, the paper does not provide sufficiently targeted evaluations of identity persistence, reference ambiguity, or cross-turn trajectory consistency to establish the distinct benefits of TRK.
Questions for authors
  • Given the relatively small accuracy gains on Dyn-Bench, how do the authors justify the additional architectural and training complexity?
  • How well does the method generalize to independently annotated trajectory-grounded dialogues and unseen scene distributions?
Reviewer_HAv8OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a task, dataset, and method for tracking objects in 3D space over time (4D). This can be seen as a way of merging 2 tasks: object tracking and video QA by providing a dataset with shared labels and a method to operate over them.

Critical strengths

The paper proposes a benchmark for trajectory-grounded dialogue. This is both a task + dataset contribution. The data seems to be useful (3D timestamped trajectories), and including video QA can serve as a solid benchmark for understanding the 3D perceptive capabilities of generalist AI systems.

The paper's proposed modeling approach is also coherent and well-supported by the experiments in the paper. By connecting geometric + temporal encoding with persistent entity representations and the trajectory decoder the authors show increases in performance. They have validated their training recipe + geometric decoding approach.

Critical weaknesses

This work feels a bit like combining two distinct tasks (object tracking + video QA) into one dataset. I think by itself, this is not necessarily problematic. However, I feel as though there is a lack of analyses about what kinds of capabilities this enables. Concretely, why should we care about combining these two tasks together? And what capabilities benefit from this combination? The current evaluations mainly establish improvements for either language or geometric outputs rather than looking at which joint capabilities the combination better enables.

Questions for authors

Which capabilities are enabled or improved by the combination of skills proposed in this work? Are there specific types of questions or situations where joint capabilities is necessary, or even improves performance beyond what two separate single tasks models could do?

1 comment

Dear Reviewer HAv8,

We would like to express our sincere gratitude for the time and care you devoted to reviewing our paper and for your valuable feedback. With less than two days remaining in the discussion period, we would be very grateful if you could take a moment to review our comments.

Please let us know if any concern remains unresolved or if any part of our response would benefit from further clarification. We would greatly appreciate your feedback and would be happy to address any remaining questions before the discussion closes.

Thank you again for your time and consideration.

Reviewer_a2XnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulate multi-turn trajectory-grounded spatio-temporal QA task. The paper 1) introduce Track4D-Bench, which is constructed from existing datasets, 2) propose a geometry-conditioned entity state, which helps maintain the conversation status, 3) conduct experiments and show positive results.

Critical strengths
  1. The paper proposes multi-turn dialogue for trajectory spatiotemporal grounding.
  2. The paper present new architecture for the task and verifies its advantage.
Critical weaknesses
  1. The novelty of the task and benchmark is not clear. In figure 2 (b), there are five types of dialogue questions. They look like just a multi-turn version for trajectory and spatio-temporal grounding.

  2. The trajectory evaluation is not clear. From figure 1, a curve trajectory is represented by two points, which is ambiguious. How do you calculate Traj-Acc on it? I am concerned about the correctness of the evaluation metrics and quality control of reference answer.

  3. The challenge addressed by the proposal is not clear. The paper integrates several components but misses their movitations. I do not see the advantage over straightforward method: directly using the history conversation instead of maintaining a geometry state for the conversation.

Questions for authors

Please see weakness.

1 comment

Dear Reviewer a2Xn,

We would like to express our sincere gratitude for the time and care you devoted to reviewing our paper and for your valuable feedback. With less than two days remaining in the discussion period, we would be very grateful if you could take a moment to review our comments.

Please let us know if any concern remains unresolved or if any part of our response would benefit from further clarification. We would greatly appreciate your feedback and would be happy to address any remaining questions before the discussion closes.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.