- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces KiRoSe, a framework for segmenting robot demonstrations and generating action descriptions from kinematic data. The method first trains a motion encoder using subtask-transition supervision, then applies temporal pooling and K-means quantization to obtain Sparse Motion Tokens. A language model is subsequently adapted to produce temporally grounded action descriptions from these tokens and the task instruction. Experiments demonstrate improved segmentation on the training embodiment, transfer to another robot and human-hand trajectories, and progress estimation under action reversal using the frozen encoder and a lightweight prediction head.
Critical strengths
The problem is practically relevant, and the representation design is well aligned with the task. Using compact kinematic representations to analyze demonstration structure provides a useful way to exploit existing robot data. Training the representation with subtask-transition supervision also directly addresses the requirements of temporal segmentation.
The segmentation improvements are supported by empirical results. In Table 2, KiRoSe achieves a Boundary F1 of 0.63 at a one-second tolerance, compared with approximately 0.41–0.42 for the alternative motion tokenizers, alongside clear improvements in Grounded F1. The cross-robot experiments provide further evidence of the learned representation’s transferability.
Critical weaknesses
The independent contribution of the discrete SMT design is not sufficiently established. The continuous-feature-plus-MLP baseline in the main text changes both the representation and the reader relative to the full model, so it does not isolate the effect of discretization. A matched comparison that directly feeds continuous features into the same language reader, using the same encoder and temporal pooling, is not presented in the main text. Consequently, it remains difficult to determine how much of the improvement comes from transition supervision, the discrete motion vocabulary, or the language-model reader. This limits the assessment of the core methodological contribution.
The contribution of the motion representation to action semantics remains insufficiently characterized. Grounded F1 provides valid evidence of improved joint segmentation and description quality, but its gains may partly result from more accurate temporal localization and do not independently quantify improvements in semantic recognition. The open-vocabulary experiment primarily supports the preservation of the language model’s existing knowledge, while the relevant words may also be inferred from the task name. The current analysis therefore does not sufficiently establish how much additional information in the action descriptions comes from the motion tokens of the specific trajectory.
Questions for authors
Have you compared discrete SMTs with directly projecting continuous features into the language model while keeping the encoder, temporal pooling, and language reader fixed? Under comparable training and input budgets, how do the two approaches compare in segmentation performance and inference cost?
When ground-truth temporal segments are fixed, or temporal matching quality is otherwise controlled, do motion tokens still substantially improve action descriptions? The main text already discusses the effects of removing or replacing motion tokens on boundary metrics. Could you also report their effects on semantic metrics and compare against using only the task instruction?
The introduction reports a Boundary F1 of 0.61 on AgiBot at a half-second tolerance, whereas Section 4.3 reports 0.40. Do these values correspond to different evaluation subsets or settings? Please clarify the evaluation protocol associated with each result.