acceptodds
Withdrawn from ICLR 2027

Encoder-Free Human Motion Understanding via Structured Motion Descriptions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The world knowledge and reasoning capabilities of text-based large language models (LLMs) are advancing rapidly, yet current approaches to human motion understanding, including motion question answering and captioning, have not fully exploited these capabilities. Existing LLM-based methods typically learn motion-language alignment through dedicated encoders that project motion features into the LLM's embedding space, remaining constrained by cross-modal representation and alignment. Inspired by biomechanical analysis, where joint angles and body-part kinematics have long served as a precise descriptive language for human movement, we propose Structured Motion Description (SMD), a rule-based, deterministic approach that converts joint position sequences into structured natural language descriptions of joint angles, body part movements, and global trajectory. By representing motion as text, SMD enables LLMs to apply their pretrained knowledge of body parts, spatial directions, and movement semantics directly to motion reasoning, without requiring learned encoders or alignment modules. We show that this approach goes beyond state-of-the-art results on both motion question answering (66.7% on BABEL-QA, 90.1% on HuMMan-QA) and motion captioning (R@1 of 0.584, CIDEr of 53.16 on HumanML3D), surpassing all prior methods. SMD additionally offers practical benefits: the same text input works across different LLMs with only lightweight LoRA adaptation (validated on 8 LLMs from 6 model families), and its human-readable representation enables interpretable attention analysis over motion descriptions. The code is available in Supplementary Materials.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Human Motion Understanding, Motion Question Answering, Motion Captioning, Large Language Models, Encoder-Free

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mdTXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Structured Motion Description (SMD), a rule-based, deterministic pipeline that converts 3D joint-position sequences into structured natural-language text. The text covers 26 ISB-convention joint angles segmented into increase/decrease/hold/repeat intervals with timestamps, plus a global pelvis trajectory. It is fed to an LLM fine-tuned with LoRA for motion QA (BABEL-QA, HuMMan-QA, re-formatted by the authors into a 10-option setting) and motion captioning (HumanML3D). The authors report state-of-the-art results without a learned motion encoder and argue that SMD lets LLMs exploit their pretrained knowledge of body parts and movement.

Critical strengths

The pipeline is simple, deterministic, and backbone-agnostic. Its domain invariance (e.g., on RGB-D-reconstructed HuMMan motions) is a practical advantage over frozen, dataset-specific motion encoders.

Critical weaknesses
  1. Limited novelty: text-mediated motion understanding is well established and closely related prior work is not discussed. The central idea of representing motion through structured natural language and handing it to a language model has been explored in several lines of work:
  • PoseScript and MotionScript already proposes a training-free, rule-based conversion of 3D motion into structured, fine-grained natural-language descriptions and uses it to bridge motion and LLM reasoning. This is the closest prior work and is conceptually almost identical to SMD.
  • GAP (Generative Action Description Prompts for Skeleton-based Action Recognition, ICCV 2023) uses LLM-generated, body-part-level action descriptions as textual knowledge for skeleton-based action recognition.
  • SINC (ICCV 2023) uses GPT-3 to reason about body-part involvement for compositional motion synthesis. KinMo, FineMoGen (HuMMan-MoGen), and FineMotion/MG-MotionLLM further establish that fine-grained, body-part-level language is an effective intermediate for motion–language tasks.

None of MotionScript, GAP, SINC, KinMo, or FineMoGen is cited; MG-MotionLLM is cited only as an encoder-based baseline, not as evidence that body-part-level language is already an effective intermediate. Against this background, the remaining contribution of SMD is a design choice: placing continuous, timestamped joint-angle text on the input side of an LoRA-tuned LLM. The joint-angle computation itself follows prior work, and the segmentation is standard smoothing plus peak/valley detection. The paper does not position SMD against these works, nor does it show that its specific design (continuous ISB angles vs. categorical posecodes or MotionScript-style descriptions) yields any measurable benefit.

  1. SMD is a “describe-then-answer” pipeline, but it is never compared against describe-then-answer baselines. As a result, the reported gains cannot be attributed to the proposed representation. Functionally, SMD first converts motion into a textual description and then lets an LLM answer from that description, much like retrieval-augmented generation. The relevant question is therefore not whether text input beats encoder input, but whether rule-generated text is better than text produced by existing motion-language models. The authors make exactly this argument in Sec. 2, claiming that caption-based pipelines (LLoVi, Socratic Models) inherit captioner biases and lose fine-grained detail. However, no experiment supports this claim. Specifically:
  • 2.1 The decisive baseline is missing. Captions from MotionGPT/MotionGPT3, or fine-grained body-part descriptions from MG-MotionLLM (m2dt), should be fed into the same Qwen2.5-7B + LoRA, and MotionScript text should be compared under the same setting. Without these, it is impossible to tell whether SMD’s advantage comes from its representation or simply from adopting a describe-then-answer pipeline on a 7B LLM. Table 15 does not address this: it compares the fine-tuned LLM’s generated text against FineMotion references, not the utility of SMD versus learned descriptions as input for downstream reasoning.

  • 2.2 Task–representation isomorphism inflates the results. BABEL-QA and HuMMan-QA ask which body part moves and in which direction, and SMD states these explicitly (e.g., “Left Hip Flexion”, “moves left 0.2m”). For these question types the task reduces largely to reading comprehension over the input. Similarly, FineMotion’s references are per-0.5s body-part movement descriptions (generated from start/end poses via PoseFix), which closely mirror SMD’s per-joint, per-interval structure. No per-question-type breakdown (action vs. body part vs. direction) is reported, so it remains unclear whether gains come from reasoning or from lookup (simple Retrieval Augmented Generation).

  • 2.3 The evaluation protocol is not rigorous enough. The 10-option QA format is defined by the authors and all QA baselines are retrained by them, so results cannot be compared with published numbers. Captioning baselines mix numbers from different papers with recomputed CIDEr on a 20–200-frame subset, and MG-MotionLLM’s CIDEr is missing. The FineMotion comparison pits a 7B model against copied numbers from a 770M model. BABEL-QA’s test set contains only 393 questions (95% CI +-4.7pp), and many ablation differences (Tables 2, 4) fall within this range; no multi-seed results are reported.

  • 2.4 The core claim contradicts the paper’s own evidence. The paper attributes SMD’s effectiveness to the LLM’s pretrained world knowledge, yet:

    • Zero-shot performance is weak (35.6/31.7 QA; captioning R@1 = 0.075).
    • Larger or stronger LLMs do not help consistently: Qwen2.5-14B (71.5/88.7) underperforms Qwen2.5-7B (73.3/91.0) on QA, and Gemma3-4B achieves higher R@1 (0.552) than Qwen2.5-7B (0.547).
    • No ablation anonymizes the semantic labels (e.g., replacing “Left Hip Flexion (raising thigh)” with “Channel 7”) or serializes raw numbers as text.

    This evidence is equally consistent with LoRA learning a mapping from hand-crafted features to labels (or any previous motion description work), which would not require any pretrained knowledge of the body.

Questions for authors
  1. How does SMD differ conceptually from MotionScript, GAP, and SINC? Can you compare SMD against MotionScript text as input under the identical Qwen2.5-7B + LoRA setting?
  2. Can you report describe-then-answer baselines in which captions or fine-grained descriptions from existing motion-language models (e.g., MotionGPT3, MG-MotionLLM m2dt) are fed into the same LLM + LoRA for both QA and captioning?
  3. Can you provide a per-question-type breakdown (action / body part / direction) on BABEL-QA and HuMMan-QA for SMD and all baselines?
Reviewer_WnD1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Structured Motion Description (SMD), which achieves encoder-free motion understanding by translating motion data into structural motion textual descriptions. The performance achieves competitive results across multiple motion QA and captioning benchmarks.

Critical strengths
  1. Simple and intuitive framework by transting motion data into structual motion descriptions.

  2. Competitive experimental results across multiple motion QA and captioning benchmarks.

  3. The paper conducts relatively comprehensive experimental analysis for the propose method.

Critical weaknesses
  1. Limited novelty. The idea of translating motion data into textual descriptions is not novel, as similar approaches have been explored in prior works such as PoseScript. The paper does not clearly establish new insights or address substantial technical challenges beyond existing methods.

  2. Unjustified modifications to evaluation protocols. The paper modifies the original QA evaluation protocols without sufficient justification. In particular, reducing the number of answer options may alter task difficulty and compromise comparability with prior methods. Results under the original evaluation protocols should also be reported to ensure a fair comparison.

  3. The experiments do not clearly establish whether the observed improvements stem from the semantic properties of natural language or simply from the structured representation of motion information. Comparisons with alternative representations, such as joint-angle sequences and symbolic motion descriptions, using the same LLM backbone would help isolate the specific contribution of SMD.

  4. The substantial performance gap between zero-shot inference and LoRA fine-tuning raises questions about whether the model genuinely leverages pretrained motion-related knowledge or primarily learns task-specific mappings between structured representations and motion information during fine-tuning. Further analysis is needed to disentangle these effects.

  5. Although the paper emphasizes the human-readable nature of the proposed representation, the structured motion descriptions are not necessarily intuitive to interpret. Understanding them often requires considerable spatial imagination to reconstruct the corresponding body movements, which weakens the claimed advantage of human readability.

  6. The proposed structured representation may struggle to capture complex, coordinated movements. For example, a somersault can be concisely described using a single semantic term, whereas representing it through structured descriptions requires specifying intricate, coordinated movements across multiple body parts. Converting such motions into SMD may lose important aspects of their holistic motion structure, raising concerns about the expressiveness and scalability of the proposed representation.

Questions for authors

The questions are incorporated into the summary of weaknesses.

Reviewer_riq4OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose Structured Motion Description (SMD), an encoder-free approach to human motion understanding. The method uses deterministic rules to convert joint-position sequences into structured text describing joint-angle changes, body-part movements, and global trajectories. These descriptions are fed directly to language models, with task-specific LoRA fine-tuning for motion question answering and captioning. Experiments on BABEL-QA, HuMMan-QA, and HumanML3D show improved QA accuracy and gains on most captioning metrics over the evaluated baselines. Tests across eight language models further demonstrate that the same text representation can be reused with different backbones through LoRA adaptation.

Critical strengths
  1. The framework converts motion into human-readable descriptions, providing a simple interface for language models.
  2. Experiments across three datasets show improved motion QA accuracy and captioning performance over the evaluated baselines.
Critical weaknesses
  1. The method segments motion using peaks, troughs, and change magnitudes, without explicitly identifying semantic action boundaries. The LLM must then infer relationships between actions from timestamped descriptions, but its ability to understand multi-action combinations is not sufficiently evaluated.
  2. The ablations examine joint coverage, trajectory information, and conversion thresholds, but do not compare alternative representations that preserve the same motion information.
  3. Table 14 suggests that generalization to new action distributions remains heavily dependent on target-domain supervision.
  4. Qualitative QA and captioning examples are limited, and no human evaluation is reported.
Questions for authors
  1. Have the authors compared SMD, rendered motion frames or videos, and their combination using the same vision-language backbone?
  2. How well do general-purpose models such as Gemini 4 Argon, GPT-6 Astra, or Opus 5.5 perform on rendered motion videos or frames without motion-specific fine-tuning? How do their results compare with those of specialized models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.