acceptodds
Under review as a conference paper at ICLR 2027

KiRoSe: Using Kinematic Data for Efficient Segmentation of Robot Demonstrations

Abstract

Segmenting robot demonstration data into disjoint temporal and semantic subsegments is valuable for learning, cleaning, retrieval, and analysis. Segmentation is often performed manually in a tedious and error-prone process. Multimodal Large Language Models (MLLMs) have the potential for automating this process, but directly ingesting raw multi-camera video streams into MLLMs can be costly and can quickly exhaust context windows. In this paper, we introduce KiRoSe (Kinematics for Robot Segmentation), a novel framework that focuses on the companion stream of robot kinematic data. Inspired by the principle of classical entropy coding, such as Huffman coding, where frequently occurring symbols are represented more compactly, we propose Sparse Motion Tokens (SMTs), a discrete kinematic vocabulary designed to compactly represent recurring, temporally localized motion patterns, which we refer to as kinematic events. SMTs are learned by encoding kinematic trajectories with temporal-segment supervision and quantizing the resulting features. We use increasingly available robot demonstration subtask annotations to supervise an open language model to interpret the resulting token sequences. Evaluation against human-labeled segmentation shows that KiRoSe can substantially outperform baseline segmentation methods in F1 quality and transfers zero-shot to unseen robot embodiments and ego-centric human motion, while preserving the language model's open-vocabulary knowledge. Beyond segmentation, KiRoSe tracks task progress on an unseen robot and detects when an action is being reversed.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot demonstration segmentation, Motion tokenization, Robot kinematics, Multimodal large language models, Temporal action understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vhunOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces KiRoSe, a framework for segmenting robot demonstrations and generating action descriptions from kinematic data. The method first trains a motion encoder using subtask-transition supervision, then applies temporal pooling and K-means quantization to obtain Sparse Motion Tokens. A language model is subsequently adapted to produce temporally grounded action descriptions from these tokens and the task instruction. Experiments demonstrate improved segmentation on the training embodiment, transfer to another robot and human-hand trajectories, and progress estimation under action reversal using the frozen encoder and a lightweight prediction head.

Critical strengths

The problem is practically relevant, and the representation design is well aligned with the task. Using compact kinematic representations to analyze demonstration structure provides a useful way to exploit existing robot data. Training the representation with subtask-transition supervision also directly addresses the requirements of temporal segmentation.

The segmentation improvements are supported by empirical results. In Table 2, KiRoSe achieves a Boundary F1 of 0.63 at a one-second tolerance, compared with approximately 0.41–0.42 for the alternative motion tokenizers, alongside clear improvements in Grounded F1. The cross-robot experiments provide further evidence of the learned representation’s transferability.

Critical weaknesses

The independent contribution of the discrete SMT design is not sufficiently established. The continuous-feature-plus-MLP baseline in the main text changes both the representation and the reader relative to the full model, so it does not isolate the effect of discretization. A matched comparison that directly feeds continuous features into the same language reader, using the same encoder and temporal pooling, is not presented in the main text. Consequently, it remains difficult to determine how much of the improvement comes from transition supervision, the discrete motion vocabulary, or the language-model reader. This limits the assessment of the core methodological contribution.

The contribution of the motion representation to action semantics remains insufficiently characterized. Grounded F1 provides valid evidence of improved joint segmentation and description quality, but its gains may partly result from more accurate temporal localization and do not independently quantify improvements in semantic recognition. The open-vocabulary experiment primarily supports the preservation of the language model’s existing knowledge, while the relevant words may also be inferred from the task name. The current analysis therefore does not sufficiently establish how much additional information in the action descriptions comes from the motion tokens of the specific trajectory.

Questions for authors

Have you compared discrete SMTs with directly projecting continuous features into the language model while keeping the encoder, temporal pooling, and language reader fixed? Under comparable training and input budgets, how do the two approaches compare in segmentation performance and inference cost?

When ground-truth temporal segments are fixed, or temporal matching quality is otherwise controlled, do motion tokens still substantially improve action descriptions? The main text already discusses the effects of removing or replacing motion tokens on boundary metrics. Could you also report their effects on semantic metrics and compare against using only the task instruction?

The introduction reports a Boundary F1 of 0.61 on AgiBot at a half-second tolerance, whereas Section 4.3 reports 0.40. Do these values correspond to different evaluation subsets or settings? Please clarify the evaluation protocol associated with each result.

Reviewer_LoEVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KiRoSe, a framework for segmenting robot demonstrations using kinematics and task instructions. A temporal encoder is trained with subtask-boundary supervision, then frozen; uniformly pooled features are quantized into a 512-entry motion vocabulary. A LoRA-adapted language model reads these tokens with timestamps and generates segment boundaries and natural-language descriptions. The paper evaluates held-out segmentation, cross-embodiment transfer, vocabulary retention, and a separate progress-estimation probe.

The central contribution is a compact, task-aligned motion representation for demonstration understanding. The segmentation results are promising, but the experiments do not fully isolate the effects of transition supervision, discretization, and language-model decoding.

Critical strengths
  • The proposed pipeline is technically plausible, and the same-reader tokenizer comparisons provide useful evidence.
  • Transition-supervised motion compression is a sensible contribution with potential value for robot-data annotation.
Critical weaknesses
  • The central contribution is not sufficiently isolated. Table 2 shows continuous features plus MLP outperforming quantized features plus MLP, while the complete system benefits from the language-model reader. A comparison feeding the same pooled continuous features into the same LLM would clarify whether discretization provides a favorable accuracy–efficiency trade-off. A matched encoder-objective comparison would further isolate transition supervision.
  • Key interpretations require better controlled experiments: Grounded F1 does not isolate semantic quality from boundary accuracy, while the progress probe combines continuous encoder features with accumulated displacement.
  • The introduction reports AgiBot Boundary F1 of 0.61, whereas section 4.3 reports 0.40. This materially affects the headline result and needs to be revised.
  • Limited validation across language-model readers. All main tokenizer comparisons use a single LoRA-adapted Qwen3.5-9B backbone 。
Questions for authors

Which AgiBot result is correct, 0.61 or 0.40, and are comparison methods evaluated on identical subsets?

Reviewer_2pVxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose KiRoSe, a framework for robot demonstration segmentation using motion data instead of videos. the main idea is to learn compact representations from subtask transitions rather than reconstructing motion signals. It introduces sparse motion tokens which is learnt from boundary supervised encoding. the sparse motion tokens are then interpreted by an adapted language model to predict subtask boundaries and action descriptions. Experiments show that KiRoSe outperforms existing methods on segmentation, generalises to unseen robots, and capture useful information for task progress estimation and action reversal detection.

Critical strengths

I think the paper has the following strengths:

  1. Clear motivation and intuitive approach. Using motion data instead of video is a practical and more generalisable way to reduce computational costs. Learning from subtask transitions rather than motion reconstruction is also simple and well-motivated idea.
  2. Strong results. KiRoSe outperforms existing baselines like FAST, OAT and MotionGPT.
  3. Generalisation across robots. the method has been tested on unseen robots and human-hand trajectories without additional training. the results suggests that the learned motion tokens can generalise across different embodiments to some extent.
Critical weaknesses

Thank you for the awesome work! i have the following concerns and hope the authors may address:

  1. lack of comparison with segmentation oriented baselines. the baseline comparisons are not entirely sufficiently, as existing motion tokenizers are mainly for reconstruction rather than for specific task like KiRoSe. Comparing against segmentation-focused method or finetuned VLM would better demonstrate whether KiRoSe's improvements come from its representation or from the additional supervisions.
  2. Insufficient ablation of the proposed motion tokenization. table 2 shows continuous features outperform discrete SMT codes with an MLP. although SMT+LLM performs the best, but it is unclear if the gain is from the better LLM or it is from SMT. A comparison using the same LLM with latent would help clarify this.
Questions for authors

I have the following questions and hope the authors may answer:

  1. could the authors compare KiRoSe against methods specifically designed for trajectory segmentation or fine-tuned VLMs, to better show whether the improvements come from proposed representation rather than task-specific supervision?
  2. could the authors compare discrete SMT tokens with latent features using the same LLM reader, to better isolate the benefit of the proposed tokenisation scheme?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.