- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper demonstrates that a pretrained VLM can be fine-tuned to directly autoregress a structured, object-centric 6-DoF motion representation, providing an embodiment-independent alternative to dedicated trajectory decoders. It augments token cross-entropy with a typical numerical regression objective and uses bidirectional motion-language training to derive a cycle-reconstruction error that serves as an inference-time confidence measure. None of these contributions is particularly innovative. Probably the largest contribution is described in the appendix: the data generation pipeline that runs a large set of existing methods to get the training labels for the fine-tuning. This substantial, labeled dataset most likely explains the improvement over baselines.
Critical strengths
Using a large zoo of existing methods for segmentation, tracking, depth estimation, etc. a comparatively strong automatic labeling of 150k samples is achieved, which provides a strong basis for fine-tuning the VLM.
Critical weaknesses
The paper is written in very vague terms from the beginning to the end rather than saying precisely what it is doing, namely simply fine-tuning a VLM for object-trajectory prediction without using additional generative components. The argument why the latter is "bad" is weak and without evidence. There is a lot of framing happening with the wording, which leads to overclaiming, for example "hierarchical representation", which is simply a sequence of the defined steps "predict relevant objects + bbox/category/role; predict current and desired object poses; predict intermediate poses". Directly generating spatial plans from a VLM is established in the field. The paper does not explicitly claim any novelty, but the writing frames it as if the paper was full of new ideas. It is not. In fact, I did not find any substantial novelty that is of scientific value in this paper.
Questions for authors
No questions.