acceptodds
Under review as a conference paper at ICLR 2027

From Behavior to Kinematics: A Hierarchical Motion-Language Benchmark for Rodent Motion Generation

Abstract

Mice and rats are central to experimental studies of brain function and behavior, where brief changes in posture and local movement carry information that broad activity labels can miss. Studying language-conditioned generation of rodent motion requires motion–language pairs that preserve these observed details. We introduce RodentBEK, a benchmark built from real mouse and rat recordings with a shared 25-landmark skeleton and 8,195 motion windows. Its Behavior–Event–Kinematics (BEK) framework connects whole-window behavior captions, half-second event descriptions, and pose-derived kinematic evidence. Events are described from video and summarized into behavior captions; kinematic measurements support verification and revision of movement claims. The benchmark contains 24,585 behavior captions and 46,679 nonempty event descriptions, with source-time-separated training, validation, and test partitions. We evaluate diffusion, autoregressive, and masked-token generators with species-specific motion–text evaluators, comparing behavior-only and behavior-plus-event inputs to the same trained model. Event conditioning improves local event correspondence across all three models and both species, while its effects on global behavior alignment and motion distributions depend on the model. RodentBEK provides a benchmark for studying how generative models use fine-grained language to represent rodent movement.

Primary area:
datasets and benchmarks
Keywords:
Text-to-Motion Generation, Animal Motion Generation, Motion-Language Benchmark, Rodent Behavior

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Z27YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper constructs a novel motion-language benchmark for mice and rats, the first dataset that contains both behavior- and event-level descriptions together with kinematic information. It further trains three representative text-to-motion models on the constructed dataset using descriptions of different levels, showing the usefulness of this dataset.

Critical strengths

This is the first rodent motion-language datset to have both behavior- and event-level descriptions together with kinematic information.

Critical weaknesses

Motivation: While the paper mentions that "Mice and rats are widely used to investigate how neural activity, genetic variation, and experimental interventions relate to behavior. Posture, locomotion, exploration, and grooming reveal responses to the environment", this work does not show how this dataset can be used for related downstream tasks. Instead this work focuses on text-to-motion generation, a classical task in computer vision. This raises concerns about this paper's motivation.

Questions for authors

Could you elaborate on your motivation for this work, especially how this dataset or text-to-motion tasks can be used for relevant downstream tasks or scientific exploration? While I am not an expert, I guess this might be related to neuroscience or cognitive science.

Reviewer_xWR9OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors have introduced RodentBEK, a benchmark for rodent motion-language models. It uses real laboratory videos and links whole-window Behavior captions, 0.5-second Event descriptions, and physical Kinematic measurements. They tested 3 baseline models (MDM, T2M-GPT, and MoMask) by comparing inputs with only Behavior text against inputs with both Behavior and Event text.

Critical strengths
  • The authors provide an annotated dataset with practical applications for cross- disciplines, which extends the boundaries of Machine Learning and Computer Vision to other research areas.

  • The proposed pipeline effectively utilises Kinematics (K) to verify and correct potential hallucinations generated during the VLM process.

Critical weaknesses
  • The pipeline relies on existing datasets (e.g., Rat-7M) and a pose estimation algorithm (Li et al., 2023) without adding new original data/algorithm. This raises concerns regarding whether the technical contribution is enough for an ICLR publication.

  • The 0.5-second sampling and variable-length windows for Event (E) extraction sounds like a reasonable approach to handle event length variation. However, it remains unclear how the proposed 2-4 second window range actually captures semantically meaningful event clips if different rodents perform the same behaviours at varying speeds.

  • The T2M generation methods surveyed in Section 2.3 and evaluated in Section 4.1 are entirely based on human motion generation. The authors sis not consider recent T2M methods designed for animal motions such as: OmniMotionGPT [Yang et al., 2024], OmniZoo [Chen et al., 2025], SAMoR [Zhang et al., 2026], NECromancer [Xu et al., 2026], and AnyMoLe [Yun et al., 2025].

  • In Table 3, the performance difference between B-only and B+E inputs is inconsistent across the baseline models. It is unclear whether this is from architectural limitations of the models or any issues in the dataset. If the dataset itself causes these model-dependent performance inconsistencies, it can be a critical flaw. Additionally, the gains from adding Event (E) conditioning are marginal (within the margin of error).

  • The paper does not mention any plans for the public release of the dataset.

Questions for authors
  • Clarify the main message and core technical contributions of this work for this community “International Conference on Learning Representations” (not for rodent behaviour research).

  • Provide a more detailed explanation of the way to determine the exact start times and lengths when extracting the variable-length events E.

  • Explain why recent T2M architectures designed for animal motion generation were not considered in the benchmark evaluations.

  • Provide a deeper analysis of the results presented in Table 3, especially, whether the inconsistent and marginal performance shifts between B-only and B+E inputs hold practical significance.

  • There are ways to share data, demos and samples during double-blind reviews (e.g., Anonymous GitHub, Zenodo). Do you intend to release this dataset publicly in a very structured way?

Reviewer_weHDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RodentBEK, a dataset of 8,195 reconstructed mouse and rat motion clips from laboratory recordings. Each clip has a behavior description and half-second event descriptions, with kinematic measurements used to check the annotations. Experiments with three generators show that adding event descriptions improves local matching scores in both species, while global scores show mixed results.

Critical strengths
  1. The dataset provides real laboratory rodent motion paired with descriptions at both clip and event levels. The shared skeleton gives mouse and rat data a common format.
  2. The experiments compare B and B+E inputs using the same model weights. Local scores improve across three generators and both species, showing that the dataset supports a useful study of detailed text conditioning.
  3. The paper includes a blinded caption-quality study and tests whether the evaluator responds to mismatched descriptions, shifted events, and static motion. These checks support the dataset and evaluation setup.
Critical weaknesses
  1. The local evaluator has been tested on altered reference motions, but its scores have not been checked against independent judgments of generated motions. Higher local scores may not always mean that small movements are more correct.
  2. The human study focuses on whole-clip captions. The accuracy of individual event descriptions and revisions made using kinematic evidence needs further checking.
  3. Training and test sets can contain the same animals and sessions. The results therefore do not establish performance on new animals or recording conditions.
Questions for authors
  1. Do the local scores agree with human judgments or direct kinematic checks of generated motions?
  2. Can you report the error rate of event annotations and how often kinematic verification fixes, misses, or introduces an error?
  3. Does the improvement from B+E hold across individual recordings and independently trained evaluators?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.