- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Structured Motion Description (SMD), a rule-based, deterministic pipeline that converts 3D joint-position sequences into structured natural-language text. The text covers 26 ISB-convention joint angles segmented into increase/decrease/hold/repeat intervals with timestamps, plus a global pelvis trajectory. It is fed to an LLM fine-tuned with LoRA for motion QA (BABEL-QA, HuMMan-QA, re-formatted by the authors into a 10-option setting) and motion captioning (HumanML3D). The authors report state-of-the-art results without a learned motion encoder and argue that SMD lets LLMs exploit their pretrained knowledge of body parts and movement.
Critical strengths
The pipeline is simple, deterministic, and backbone-agnostic. Its domain invariance (e.g., on RGB-D-reconstructed HuMMan motions) is a practical advantage over frozen, dataset-specific motion encoders.
Critical weaknesses
- Limited novelty: text-mediated motion understanding is well established and closely related prior work is not discussed. The central idea of representing motion through structured natural language and handing it to a language model has been explored in several lines of work:
- PoseScript and MotionScript already proposes a training-free, rule-based conversion of 3D motion into structured, fine-grained natural-language descriptions and uses it to bridge motion and LLM reasoning. This is the closest prior work and is conceptually almost identical to SMD.
- GAP (Generative Action Description Prompts for Skeleton-based Action Recognition, ICCV 2023) uses LLM-generated, body-part-level action descriptions as textual knowledge for skeleton-based action recognition.
- SINC (ICCV 2023) uses GPT-3 to reason about body-part involvement for compositional motion synthesis. KinMo, FineMoGen (HuMMan-MoGen), and FineMotion/MG-MotionLLM further establish that fine-grained, body-part-level language is an effective intermediate for motion–language tasks.
None of MotionScript, GAP, SINC, KinMo, or FineMoGen is cited; MG-MotionLLM is cited only as an encoder-based baseline, not as evidence that body-part-level language is already an effective intermediate. Against this background, the remaining contribution of SMD is a design choice: placing continuous, timestamped joint-angle text on the input side of an LoRA-tuned LLM. The joint-angle computation itself follows prior work, and the segmentation is standard smoothing plus peak/valley detection. The paper does not position SMD against these works, nor does it show that its specific design (continuous ISB angles vs. categorical posecodes or MotionScript-style descriptions) yields any measurable benefit.
- SMD is a “describe-then-answer” pipeline, but it is never compared against describe-then-answer baselines. As a result, the reported gains cannot be attributed to the proposed representation. Functionally, SMD first converts motion into a textual description and then lets an LLM answer from that description, much like retrieval-augmented generation. The relevant question is therefore not whether text input beats encoder input, but whether rule-generated text is better than text produced by existing motion-language models. The authors make exactly this argument in Sec. 2, claiming that caption-based pipelines (LLoVi, Socratic Models) inherit captioner biases and lose fine-grained detail. However, no experiment supports this claim. Specifically:
-
2.1 The decisive baseline is missing. Captions from MotionGPT/MotionGPT3, or fine-grained body-part descriptions from MG-MotionLLM (m2dt), should be fed into the same Qwen2.5-7B + LoRA, and MotionScript text should be compared under the same setting. Without these, it is impossible to tell whether SMD’s advantage comes from its representation or simply from adopting a describe-then-answer pipeline on a 7B LLM. Table 15 does not address this: it compares the fine-tuned LLM’s generated text against FineMotion references, not the utility of SMD versus learned descriptions as input for downstream reasoning.
-
2.2 Task–representation isomorphism inflates the results. BABEL-QA and HuMMan-QA ask which body part moves and in which direction, and SMD states these explicitly (e.g., “Left Hip Flexion”, “moves left 0.2m”). For these question types the task reduces largely to reading comprehension over the input. Similarly, FineMotion’s references are per-0.5s body-part movement descriptions (generated from start/end poses via PoseFix), which closely mirror SMD’s per-joint, per-interval structure. No per-question-type breakdown (action vs. body part vs. direction) is reported, so it remains unclear whether gains come from reasoning or from lookup (simple Retrieval Augmented Generation).
-
2.3 The evaluation protocol is not rigorous enough. The 10-option QA format is defined by the authors and all QA baselines are retrained by them, so results cannot be compared with published numbers. Captioning baselines mix numbers from different papers with recomputed CIDEr on a 20–200-frame subset, and MG-MotionLLM’s CIDEr is missing. The FineMotion comparison pits a 7B model against copied numbers from a 770M model. BABEL-QA’s test set contains only 393 questions (95% CI +-4.7pp), and many ablation differences (Tables 2, 4) fall within this range; no multi-seed results are reported.
-
2.4 The core claim contradicts the paper’s own evidence. The paper attributes SMD’s effectiveness to the LLM’s pretrained world knowledge, yet:
-
- Zero-shot performance is weak (35.6/31.7 QA; captioning R@1 = 0.075).
-
- Larger or stronger LLMs do not help consistently: Qwen2.5-14B (71.5/88.7) underperforms Qwen2.5-7B (73.3/91.0) on QA, and Gemma3-4B achieves higher R@1 (0.552) than Qwen2.5-7B (0.547).
-
- No ablation anonymizes the semantic labels (e.g., replacing “Left Hip Flexion (raising thigh)” with “Channel 7”) or serializes raw numbers as text.
This evidence is equally consistent with LoRA learning a mapping from hand-crafted features to labels (or any previous motion description work), which would not require any pretrained knowledge of the body.
Questions for authors
- How does SMD differ conceptually from MotionScript, GAP, and SINC? Can you compare SMD against MotionScript text as input under the identical Qwen2.5-7B + LoRA setting?
- Can you report describe-then-answer baselines in which captions or fine-grained descriptions from existing motion-language models (e.g., MotionGPT3, MG-MotionLLM m2dt) are fed into the same LLM + LoRA for both QA and captioning?
- Can you provide a per-question-type breakdown (action / body part / direction) on BABEL-QA and HuMMan-QA for SMD and all baselines?