- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces LMM-Track4D, a framework for trajectory-grounded multi-turn spatiotemporal dialogue. The authors construct Track4D-Bench, containing 526 video clips with calibrated geometric observations and timestamped entity trajectories. The proposed method combines Ray–Time Geometry Encoding (RTGE), a persistent target-state memory (TRK), and an Object-Slot Kinematic Residual-Anchor (OSK-RA) decoder to support entity-aware dialogue and 3D trajectory prediction. The framework is trained using joint language and geometric supervision followed by GRPO
Critical strengths
- The paper addresses trajectory-grounded multi-turn dialogue, connecting persistent entity references with temporal and geometric reasoning.
- The combination of geometric encoding, target-state memory, and trajectory decoding is clearly motivated and reasonably presented.
- Track4D-Bench provides paired dialogue and geometric supervision, enabling evaluation of both language generation and metric trajectory prediction.
Critical weaknesses
- The proposed framework primarily integrates existing techniques, including geometry-aware visual encoding, persistent object representations, attention-based evidence selection, and residual trajectory regression. Related approaches such as LLaVA-4D, Ground3D-LMM, and ObjectStream have already explored geometric reasoning, multi-turn spatial grounding, or persistent object representations. Although LMM-Track4D combines these components for trajectory-grounded dialogue, the paper does not sufficiently demonstrate a fundamentally new modeling mechanism or explain why the proposed integration provides a substantial advancement over existing methods.
- Track4D-Bench contains only 526 clips, with a substantial majority originating from KITTI. The dialogues are constructed through model-assisted generation and filtering, followed by author spot-checks.
- The main results compare LMM-Track4D, which incorporates geometry-oriented supervision, joint optimization, and GRPO, against baselines primarily trained with language SFT.
- The controlled ablations in Table 3 show that adding TRK to RTGE improves SAcc and Traj-Acc by only 0.55 percentage points each, and TAcc by 0.72 points. These modest gains are difficult to reconcile with the central emphasis on persistent cross-turn entity memory. Although the standard-objective ablation shows a larger CIDEr improvement, the paper does not provide sufficiently targeted evaluations of identity persistence, reference ambiguity, or cross-turn trajectory consistency to establish the distinct benefits of TRK.
Questions for authors
- Given the relatively small accuracy gains on Dyn-Bench, how do the authors justify the additional architectural and training complexity?
- How well does the method generalize to independently annotated trajectory-grounded dialogues and unseen scene distributions?
Dear Reviewer HAv8,
We would like to express our sincere gratitude for the time and care you devoted to reviewing our paper and for your valuable feedback. With less than two days remaining in the discussion period, we would be very grateful if you could take a moment to review our comments.
Please let us know if any concern remains unresolved or if any part of our response would benefit from further clarification. We would greatly appreciate your feedback and would be happy to address any remaining questions before the discussion closes.
Thank you again for your time and consideration.