acceptodds
Under review as a conference paper at ICLR 2027

Learning Reusable Motion Primitives by Separating Identity from Execution

Abstract

Complex skills can be viewed as organized compositions of reusable motion primitives. Learning such primitives therefore requires an explicit notion of equivalence: when do different motion trajectories instantiate the same primitive? Recent action-representation research spans action chunking, tokenization, and latent-action learning, with increasing attention to shaping latent-action semantics; yet these advances do not by themselves specify what defines reusable primitive identity. To make this equivalence explicit, we define primitive identity as the ordered structure of canonical basic motions, while execution captures how that structure is realized through local transformations, relative rhythm, and overall temporal scale. We instantiate this separation in a hierarchical model that canonicalizes local motion geometry, encodes the ordered structure of the corresponding basic motions, and estimates execution factors separately. We evaluate the resulting representation by testing cross-execution reuse, preservation of the complete identity-defining structure, and identity–execution recombination, and further test whether the model can detect and identify primitive occurrences within longer, unsegmented trajectories. On a controlled benchmark of variable-length primitives, the model recovers both primitive structure and execution from isolated recordings and recovers primitive boundaries and identities within longer trajectories composed from a known set of primitives. Two comparison settings reveal distinct gaps. A fixed-duration chunking interface is poorly aligned with the variable temporal support of primitive instances. Even with oracle primitive support, reconstruction-based models with primitive-level semantic shaping—including a factorized extension with supervised execution—do not satisfy the required representation responsibilities: cross-execution reuse, complete-structure preservation, and recombination. Together, these results show that defining primitive identity separately from execution yields a learnable representation with a directly testable criterion for recognizing the same primitive across execution variations and across occurrences within longer trajectories.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
motion primitives, action representation learning, structured representation learning, primitive identity, structure–execution factorization

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_GKavOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript focuses on motion primitive analysis. The underlying assumption is that complex motions can be understood as sequences of primitive motions. The paper asks how to determine when two motions should be regarded as the same primitive. It proposes to analyze input motion along two dimensions, identity and execution, in order to obtain primitives. The authors report that fixed-length segmentation is not effective and that reconstruction-based models, even with semantic constraints, do not perform these tasks well.

Critical strengths

The attempt to analyze motion and extract reusable primitives is valuable. It may benefit motion composition and phase analysis.

Critical weaknesses

From the abstract and introduction, I cannot identify the focus of the manuscript. Is it about the reusability of end-effector motions for robotic arms, humanoid robotic motion, or human motion analysis? The paper appears to pursue a fundamental innovation, but it still needs an application scenario. For example, when contrastive learning was proposed, it was still validated on image classification. After reading the abstract and introduction, I honestly do not know what the paper is doing.

At the methodological level, the paper only describes conceptual content. There is no concrete implementation. There is no specific application or implementation. After reading it, I do not know how the method is implemented. This is a clear characteristic of auto research/ARIS: it is full of high-level ideas and lacks low-level implementation. It is all concepts, with no explanation of how to implement them.

I understand the authors may argue that the paper contains implementation details, such as details of the network structures and the training data. They may argue that they provide the number of network layers, the amount of training and testing data, and how augmentation is performed. However, an ordinary researcher would find this strange. The paper mentions the use of GRU and Transformer structures. Given that different tasks may require modifications to the model, Table 2 does not reflect this. It is full of generic content and even contains arrows, which is an unusual and AI-like form of expression. I also cannot determine where the experimental data comes from. It is not related to motion datasets I am familiar with, such as AIST++, AMASS, or HumanML3D.

There are no existing works used for comparison. This is not consistent with standard academic practice. If this is the first pioneering work and no existing work has addressed the same problem, or if existing works have different objectives, the authors should at least adapt existing peer-reviewed published works to the objective of this paper for comparison.

From the experimental results, the representation in the qualitative results is also unusual. I do not think anyone would draw figures in this way. All these signs, including the high-level description, the unreasonable experiments, and the experimental result figures that do not look human-made, indicate that this is a clear rejection.

Questions for authors

See "Critical Weaknesses".

2 comments

We sincerely thank the reviewer for the careful review. We address the reviewer’s concerns point by point, and we hope that our clarifications below address the reviewer’s concerns.

Q1: About the focus of the manuscript.

A1: Thank you for raising these concerns. Our long-term goal is to use reusable motion primitives as units for motor planning in complex action-sequence tasks. A prerequisite is to specify what makes a primitive the same unit across different executions. This motivates the central question of this paper: when should trajectories with different executions be regarded as instances of the same primitive?

We address this question by representing each primitive instance in terms of structure and execution. Structure defines primitive identity through the ordered composition of canonical basic motions, including their geometry and associated actors; execution describes how that structure is realized. Under this definition, trajectories instantiate the same primitive when they preserve this ordered structure despite changes in execution.

Building on this definition, we seek to represent each primitive in a form that subsequent motor planning can use. The learned structural representation should therefore preserve the constituent motion geometry and order while remaining stable across the specified execution variations. The separately estimated execution factors should describe how these motions are realized through amplitude, duration, and relative rhythm. The focus of this paper is to establish these representation responsibilities, with the representations serving as an intended interface for subsequent motor planning.

Q2: About the concrete implementation and application

A2: Thank you for raising these concerns. We describe the concrete method in Sec. 3.2 and provide the step-by-step training, inference, and longer-trajectory recovery procedures in Algorithms 1–3, with architecture and training details in the appendix. If any implementation step remains unclear, we would appreciate the reviewer identifying it so that we can provide a more detailed explanation.

The intended applications follow from our central question and the complementary responsibilities of structure and execution. The learned structural representation should preserve motion geometry and constituent order across execution variations, while the execution factors capture how each instance is realized. This separation supports recognizing different executions as instances of the same primitive. By retaining the primitive’s motion content and ordered composition, the structural representation is also intended to serve as an interface for subsequent motor planning, with execution specified separately.

Q3: About the claim of auto research.

A3: We would like to clarify that this work was not conducted through autonomous research. We formulated the research questions, made the methodological and experimental decisions, and interpreted the results. We used AI tools for assistance, as disclosed in our AI Use Statement, and reviewed and validated their outputs.

Q4: About the real datasets and external comparisons.

A4: Thank you for raising these concerns. We address the real-data evaluation and comparison design below.

Real-motion evaluation. We conducted additional experiments on HDM05 [1], using its public normalized 3D skeleton release. We selected forward and backward rotations of the left and right arms, together with squats, for basic-motion evaluation, and composed recorded arm-motion segments into primitives containing 2–4 basic motions. We evaluated cross-execution retrieval, execution exchange, ordered-structure readout, and recovery in longer trajectories on held-out recordings, with Sem-AE and Sem-Factor adapted to the same motion interface. The table below reports the results of seed 0 and 1.

MethodRetrieval (R@1) ↑Retrieval (mAP) ↑Execution exchange (nRMSE) ↓Ordered-structure (exact recovery) ↑Long-sequence (joint recall) ↑
OURS1.000 / 1.0001.000 / 1.0000.2202 / 0.22141.000 / 1.0001.000 / 1.000
Sem-AE0.738 / 0.7440.529 / 0.5370.4240 / 0.42120.369 / 0.3750.796 / 0.737
Sem-Factor0.774 / 0.7200.662 / 0.5860.3736 / 0.41290.452 / 0.3270.639 / 0.514

These results support the intended structure representation and execution responsibilities on recorded human motion: our model recognizes the same primitive across different executions, its learned structural representation preserves motion geometry and constituent order, and the execution retains execution-specific variation.

Purpose and scope of comparisons. Our experiments are designed to test the feasibility of the proposed formulation and whether structure and execution fulfill their respective responsibilities. The fixed-chunk comparison tests alignment with variable primitive boundaries. The representation comparisons test whether structure preserves motion geometry and ordered composition while execution retains realization variations. Sem-AE and Sem-Factor examine whether reconstruction with primitive-level semantic shaping, including a separate supervised execution latent, suffices to meet these requirements. These comparisons assess the value of explicitly assigning different responsibilities to structure and execution, rather than establish a general ranking on a downstream task.

Direct numerical comparisons with existing primitive methods, such as EMMP[2] and VLBiMan[3], are difficult because their primitive definitions and required task interfaces differ from ours. EMMP models task-conditioned trajectory manifolds under specified task symmetries, while VLBiMan reuses object-anchored manipulation modules through vision-language grounding. Our primitive identity is defined by the ordered composition of actors and canonical motion geometries across execution variations. An aligned comparison would require task parameters and symmetry actions for EMMP, and task text, visual object/grasp information, and robot kinematics for VLBiMan. These interfaces are absent from our trajectory benchmarks, so adapting these methods would require a common task and supervision protocol.

[1] Meinard Müller, Tido Röder, Michael Clausen, Bernhard Eberhardt, Björn Krüger, and Andreas Weber. Documentation: Mocap Database HDM05. Technical Report CG-2007-2, Universität Bonn, 2007.

[2] B. Lee, Y. Lee, S. Kim, M. Son, and F. C. Park, "Equivariant motion manifold primitives," in Proceedings of the 7th Conference on Robot Learning (CoRL), vol. 229, 2023, pp. 1199–1221.

[3] H. Zhou and K. Jia, "VLBiMan: Vision-language anchored one-shot demonstration enables generalizable bimanual robotic manipulation," in International Conference on Learning Representations (ICLR), 2026.

Reviewer_JRX6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reusable motion primitives by separating primitive identity from execution. A primitive is represented as p=(s,e), where s is an ordered variable-length sequence of canonical basic motions and e contains local execution, relative rhythm, and overall temporal scale. Two primitive instances are declared equivalent exactly when their ordered structures agree. The proposed hierarchy uses a bidirectional GRU support detector, canonicalizes local motion geometry, represents geometry with cubic B-spline coefficients, assigns motions to a small learned canonical vocabulary, and encodes the ordered sequence with a Transformer to obtain a structure representation. Execution factors are estimated separately, largely analytically. The method is trained in stages with temporal-support supervision and canonical ordered-structure objectives. It is evaluated on a controlled synthetic benchmark for cross-execution retrieval, complete-structure recovery, structure-execution recombination, noise and range extensions, and closed-bank primitive recovery in longer trajectories.

Critical strengths
  • The paper is easy to follow.

  • The structure-execution decomposition yields concrete responsibilities for identity invariance, complete-structure preservation, and recombination. This is more diagnostic than reporting only reconstruction loss or downstream success. The benchmark is also deliberately constructed to isolate temporal alignment from representation quality through fixed-width and oracle-support comparisons.

  • The experiments include cross-execution retrieval, held-out ordered compositions of known basic motions, matched order-only tests, identity-execution recombination, closed-bank recovery without primitive boundaries, observation noise, execution-range extrapolation, and longer trajectories. The manuscript also reports architecture, training stages, benchmark counts, and several robustness conditions in sufficient detail to understand the intended evaluation.

Critical weaknesses
  • Primitive identity is defined by assumption and is not shown to be meaningful or transferable. The paper defines p_A equivalent to p_B whenever their ordered canonical basic-motion structures match, with actor identity and a fixed canonical vocabulary placed inside the structure. This is a modeling choice, not an empirically established notion of primitive identity. The benchmark is also generated from the same structure-execution factorization, and the proposed model receives direct ordered-structure supervision. Near-perfect recovery therefore demonstrates consistency with the authors' definition, not that the definition captures reusable behavior outside the synthetic setting. No evidence is provided for unseen actors or embodiments, new basic motions, realistic trajectories, or downstream transfer.

  • The objective is under-specified, the inference path is inconsistent with training, and the order-only evaluation is not evaluation-blind. The training objective is described only through terms L_f, L_geo, L_inv^c, L_h, L_K, and L_inv^z, none of which is defined. Readers therefore cannot determine how invariance is enforced or reproduce the representation. The training algorithm feeds E_c(c_hat_i) into the structure path, while the inference algorithm feeds E_c(c_i^S); these inputs are not equivalent, and the paper does not identify which route generated the reported results. In addition, Appendix B.3 explicitly states that calibration structures include some structures used in the matched order-only evaluation and that the protocol is not development-blind. This directly compromises the headline order-preservation result.

  • Component ablations and comparisons with the closest competing formulations are missing. The paper does not ablate ordered-structure supervision, invariance losses, actor and geometry encoders, positional encoding, canonical clustering size, inverse execution transformation, the weak slot-conditioned readout, or staged versus end-to-end training. Hard canonical assignment and direct structural supervision may themselves produce much of the claimed invariance. The comparison models are not matched in architecture, capacity, or structural supervision, while EMMP and VLBiMan, which already separate invariant primitive components from adjustable realization, are not evaluated. The experiments therefore do not establish that the proposed decomposition or its individual components are responsible for the reported performance.

Questions for authors
  • Can the authors demonstrate that the stipulated equivalence relation is meaningful and transferable? Please evaluate the learned notion of primitive identity with unseen actors or embodiments, unseen canonical basic motions, realistic sensor trajectories, or a downstream compositional task. Alternatively, provide external evidence that the ordered-structure equivalence aligns with human or task-level judgments of primitive reuse. If these tests are outside the scope, the claims about general primitive reuse should be substantially narrowed.

  • Please reconcile the training and inference paths and fully specify the objective. Is the actual structure-path input E_c(c_hat_i) or E_c(c_i^S)? Please define every term in the training objective and report the results under both encoder inputs. Please also clarify whether local amplitude and duration, relative rhythm, and temporal scale are independent execution factors or deterministic functions of the same supports and anchors.

  • Please correct the evaluation protocol for order-only recovery. Training, calibration, and evaluation structures should be disjoint, and checkpoint selection should not observe any structure used in the matched order-only test. Please rerun the 234/234 experiment with a fully blind split and report results for both trained structure seeds.

  • Please provide the missing ablations and closest external comparisons. At minimum, ablate direct structure supervision, the invariance losses, positional encoding, canonical clustering size, hard versus soft canonical assignment, actor and geometry encoders, the weak readout, and staged training. Also compare against EMMP, VLBiMan, or another invariance-based primitive representation. If a direct comparison is not meaningful, explain precisely which missing components or supervision signals make the methods incomparable.

3 comments

We sincerely thank the reviewer for the careful review and constructive feedback. We address the reviewer’s concerns point by point, and we hope that our clarifications and additional analyses below address the reviewer’s concerns.

Q1: About the evaluation on unseen settings and real datasets.

A1: We sincerely thank you for this helpful suggestion. We added tests in unseen settings and a real-motion evaluation to examine whether the proposed primitive equivalence relation is meaningful and transferable, and whether structure representation and execution fulfill their intended responsibilities.

We evaluated cross-execution retrieval, execution exchange, ordered-structure recovery, and long-sequence detection with unseen actors, canonical basic motions, and constituent orders. The table below reports results averaged over four structure seeds. These results show effective retrieval, conditional execution exchange, ordered recovery, and detection under the tested unseen settings, although complete structural decoding remains imperfect.

Unseen settingRetrieval (R@1) ↑Retrieval (mAP) ↑Execution exchange (nRMSE) ↓Complete-structure (recovery) ↑Long-sequence (joint recall) ↑
Actors1.0001.0000.00470.8461.000
canonical basic motions1.0001.0000.00550.8851.000
Constituent orders1.0001.0000.00440.9231.000

For real motion, we used the normalized 3D skeleton release of HDM05 [1]. We selected forward and backward rotations of the left and right arms and squats as basic motions, and formed primitives containing basic motions from recorded segments. We evaluated the same four tasks against Sem-AE and Sem-Factor. The table below reports the results of seed 0 and 1. On held-out recordings, our model achieves better identity retrieval, ordered recovery, detection, and lower exchange error, than these semantic baselines.

MethodRetrieval (R@1) ↑Retrieval (mAP) ↑Execution exchange (nRMSE) ↓Ordered-structure (recovery) ↑Long-sequence (joint recall)↑
OURS1.000 / 1.0001.000 / 1.0000.2202 / 0.22141.000 / 1.0001.000 / 1.000
Sem-AE0.738 / 0.7440.529 / 0.5370.4240 / 0.42120.369 / 0.3750.796 / 0.737
Sem-Factor0.774 / 0.7200.662 / 0.5860.3736 / 0.41290.452 / 0.3270.639 / 0.514

Together, these results provide empirical support for the meaningfulness and transferability of the proposed primitive equivalence relation. They also support the intended responsibilities: structure representation preserves primitive identity, motion geometry, and constituent order, while execution specifies how that structure is realized.

[1] Meinard Müller, Tido Röder, Michael Clausen, Bernhard Eberhardt, Björn Krüger, and Andreas Weber. Documentation: Mocap Database HDM05. Technical Report CG-2007-2, Universität Bonn, 2007.

Q2: About the evaluation protocol for order-only recovery.

A2: We sincerely thank you for pointing out this protocol issue. We agree that training, calibration, and evaluation structures should be disjoint, and checkpoint selection should not observe any structure used in the matched order-only test.

We reran the original matched order-only experiment with disjoint ordered structures across all three splits. The results are reported below.

ProtocolMatched pairsOrder-only recovery seed 0 (%)Order-only recovery seed 1 (%)
Structure-disjoint split234100.0100.0

We will clarify this protocol and the checkpoint-selection procedure in the revised manuscript.

Q3: About the training, inference paths and the objective.

A3: We thank the reviewer for these questions and agree that the training and inference paths, objective, and execution-factor definitions require clearer explanation.

Training and inference. During representation training, EcE_c takes c^i\hat c_i as input, and its decoder reconstructs the corresponding fixed canonical target ciSc_i^S. At inference, the model first estimates c^i\hat c_i, then assigns it to a fitted geometry group, and passes the corresponding shared ciSc_i^S through the frozen encoders. Thus, training uses Ec(c^i)E_c(\hat c_i), while deployed inference uses Ec(ciS)E_c(c_i^S). We compare both inputs across cross-execution retrieval, complete-structure recovery, structure–execution recombination, and primitive recovery in longer trajectories. The results are reported below. Both inputs achieve the same performance across these four tasks.

Encoder inputCross-execution retrieval (R@1)Complete-structure (recovery)Recombination (joint success)Long-trajectory (recovery)
Ec(c^i)E_c(\hat c_i)1.000468/4682030/2048 (99.12%)1200/1200
Ec(ciS)E_c(c_i^S)1.000468/4682030/2048 (99.12%)1200/1200

Training objective. At the basic-motion level, Lf\mathcal L_f is actor-identity cross-entropy, Lgeo\mathcal L_{\mathrm{geo}} measures relative canonical-geometry reconstruction error, and Linvc\mathcal L_{\mathrm{inv}}^c is cosine distance between geometry embeddings across matched executions. At the primitive level, Lh\mathcal L_h is masked SmoothL1 reconstruction of the ordered canonical representations, LK\mathcal L_K is constituent-count cross-entropy, and Linvz\mathcal L_{\mathrm{inv}}^z is cosine distance between representations of the same ordered structure across local executions. Each stage sums its three terms, weighting invariance by 0.10.1 and the others by 11.

Execution factors. These factors have distinct semantic responsibilities: amplitude and local duration describe each basic motion’s realization, relative rhythm describes its timing relative to neighboring motions, and temporal scale describes the primitive’s overall temporal span. Amplitude and local duration parameterize the restricted local transformation through positive scaling and temporal stretching. Temporal scale is the first-to-last anchor span, and relative rhythm consists of adjacent anchor intervals normalized by that span.

We will clarify these distinctions, and provide complete objective definitions in the revised manuscript.

Q4: About the ablations and external comparisons.

A4: We sincerely thank you for these suggestions. We added ablations covering the requested components and evaluated their effects on all four tasks. The table reports two-seed means.

ComponentAblationRetrieval R@1 / mAP ↑Exchange joint success ↑Order exact recovery ↑Long-sequence joint recall ↑
Full modelOriginal configuration (k=3, hard assignment).1.0000 / 1.00000.99121.00001.0000
Direct structure supervisionRemove ordered canonical-motion sequence reconstruction loss.1.0000 / 1.00000.23630.15380.9993
Invariance lossRemove across-execution local-geometry alignment loss.1.0000 / 1.00000.99121.00001.0000
Positional encodingRemove positional encoding from the primitive encoder.0.5525 / 0.62480.01930.00000.0030
Canonical clustering sizeUse k=2.0.8281 / 0.84240.12430.03850.0020
Canonical clustering sizeUse k=4.1.0000 / 1.00000.99121.00001.0000
Hard/soft canonical assignmentReplace hard assignment with distance-weighted canonical geometry.1.0000 / 1.00000.99121.00001.0000
Actor encoderReplace the learned encoder with fixed one-hot vectors.1.0000 / 1.00000.99121.00001.0000
Geometry encoderReplace the learned encoder with standardized geometry coefficients.1.0000 / 1.00000.99121.00001.0000
the Weak readoutReplace the nonlinear slot readout with a linear head during training.1.0000 / 1.00000.43240.17310.9988
Staged trainingTrain the local-motion and primitive encoders jointly.1.0000 / 1.00000.99121.00001.0000

Removing ordered-structure reconstruction or positional encoding, or using a linear training readout, reduces complete-structure recovery and execution-exchange success. These drops support the need for them to maintain the full model’s recovery performance in this setting. With k=2k = 2, distinct local-motion geometries merge, creating ambiguous primitive identities and degrading performance.

The remaining variants preserve the task scores. The unchanged result after removing local-geometry invariance may reflect the stable canonical inputs and retained ordered-structure supervision, which reduce the additional benefit of this regularizer. Fixed encodings retain the actor and geometry information needed by these tasks. Soft assignment is already nearly one-hot, while k=4 preserves distinguishable local geometries. Joint training reaches the same scores with more updates to the local-motion encoder, so staged training achieves these scores with fewer such updates.

For external comparisons, EMMP[1] models task-conditioned trajectory manifolds using specified task symmetries, while VLBiMan[2] reuses object-anchored manipulation modules through vision-language grounding. Our identity is defined by actor identities, canonical motion geometries, and their constituent order across execution variations. An aligned numerical comparison requires a shared task parameterization, group actions, and canonical section for EMMP, and task text, visual object/grasp information, and robot kinematics for VLBiMan. These task and scene interfaces are absent from our trajectory benchmarks.

[1] B. Lee, Y. Lee, S. Kim, M. Son, and F. C. Park, "Equivariant motion manifold primitives," in Proceedings of the 7th Conference on Robot Learning (CoRL), vol. 229, 2023, pp. 1199–1221.

[2] H. Zhou and K. Jia, "VLBiMan: Vision-language anchored one-shot demonstration enables generalizable bimanual robotic manipulation," in International Conference on Learning Representations (ICLR), 2026.

Reviewer_mVZwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a hierarchical representation of reusable motion primitives that separates primitive identity from execution. Identity is defined by the ordered sequence of canonical basic motions, while execution captures local amplitude and duration, relative rhythm, and overall temporal scale. The system combines support detection, canonical geometry assignment, a learned structure encoder, and analytic execution estimation. It also recovers primitive occurrences in longer trajectories using a known primitive bank. On a controlled synthetic benchmark, the method reports perfect cross-execution retrieval and closed-bank sequence recovery, together with 99.1% structure–execution recombination success. The central question is whether these results establish a benefit of the learned representation beyond the explicit canonicalization, structural supervision, and dictionary parsing built into the system.

Critical strengths

The manuscript is generally transparent about its scope.

Critical weaknesses

The experiments do not isolate the contribution of the learned structure representation from the canonicalization frontend. In Algorithm 2, each local observation is replaced by its assigned canonical geometry before entering the structure encoder. Consequently, correctly recovered executions of the same primitive produce exactly the same ordered input tokens. Cross-execution invariance is therefore substantially enforced before the learned encoder is applied. A symbolic sequence representation using the same frontend is an essential missing baseline. In this benchmark, an ordered, padded one-hot representation would require only 45 dimensions, compared with the proposed 64-dimensional embedding, while directly preserving constituent identity, order, and length. Its end-to-end performance should be measured to establish what the learned embedding adds.

The comparison models differ in supervision, preprocessing, and training coverage, making the large performance gaps difficult to attribute. The proposed encoder directly learns to reconstruct ordered canonical constituents and benefits from order augmentation. Sem-AE and Sem-Factor instead operate on raw trajectories and receive primitive-level semantic supervision, without equivalent constituent-structure supervision during encoder training. Their structural readouts are subsequently trained on frozen representations, whereas the proposed encoder is trained jointly with its structural readout. The recombination comparison additionally contrasts an explicit structure-and-execution realization route with a learned trajectory decoder. Appendix B.3 acknowledges these differences, but matched comparisons are still needed to identify the contribution of the proposed representation design.

The benchmark is narrowly aligned with the model’s assumptions. It contains three observed actors, three canonical geometries, and primitives of two to five constituents. The main execution variations admit direct amplitude and duration normalization. Held-out structures are new combinations of the same nine basic-motion identities, rather than new motion geometries or actors. The additional stress tests are useful, but remain within this vocabulary; the noise experiments also use a separately trained support detector. The paper therefore provides limited evidence that the formulation remains effective when motion shapes vary continuously, constituent motions overlap, or canonicalization is imperfect.

The identifiability of primitive boundaries in the closed-bank experiment is unclear. Algorithm 3 chooses the longest accepted candidate whose remaining suffix can be parsed. However, a known primitive bank can admit multiple valid segmentations of the same basic-motion sequence. For example, a sequence might correspond either to two shorter primitives or to one longer primitive. The longest-match rule selects a parsing convention, but does not by itself establish recovery of the intended boundaries. The paper should explain whether the bank is uniquely decodable, whether ambiguous concatenations are excluded, and how frequently alternative parses occur in the evaluation trajectories.

The description of the training pipeline needs clarification. Algorithm 1 describes obtaining training observations through the learned support detector, whereas Appendix B.1 states that representation training used saved coefficients extracted from annotated constituent supports and was not retrained through the current raw-trajectory frontend. This distinction matters for understanding which components were trained with ground-truth supports and which were exposed to predicted-support errors. The reported implementation should be distinguished clearly from the general algorithm.

Questions for authors

N/A

2 comments

We sincerely thank the reviewer for the careful review and constructive feedback. We address the reviewer’s concerns point by point, and we hope that our clarifications and additional analyses below address the reviewer’s concerns.

Q1: About the learned structure representation.

A1: We sincerely thank you for raising this insightful concern. Our goal is to learn a hierarchical representation of motion. At the basic-motion level, the representation should retain actor identity and canonical local-motion geometry. At the primitive level, the learned structure representation should encode both the constituent basic-motion information and its sequential order. This representation is intended to provide a computational interface for future motion planning, so primitive classification alone does not capture its full intended role.

A one-hot representation can identify a basic-motion category and, when arranged sequentially, preserve category order. However, without an external dictionary storing the corresponding motion curves, these codes do not specify the underlying motion geometry. A fixed categorical vocabulary also cannot directly encode previously unseen local geometries. Thus, it does not meet the intended geometric responsibilities.

We will clarify the distinction between the canonical information supplied by the frontend and the information retained by the learned representation.

Q2: About the comparison baselines.

A2: We sincerely thank you for this insightful comment. Our comparisons address two successive questions. First, fixed-duration chunks exhibit substantial mismatch with primitive supports in our benchmark. We therefore provide oracle primitive boundaries in subsequent comparisons to isolate representation quality from temporal-support mismatch.

Second, with primitive boundaries given, we test whether the representations fulfill their intended responsibilities: structure must preserve primitive identity, constituent motion geometry, and order, while execution factors must describe how that structure is realized. Sem-AE tests whether reconstruction and primitive-level semantic losses are sufficient; Sem-Factor further tests whether separating structure and execution latents, with execution-factor supervision, achieves these responsibilities. The tested baselines do not consistently satisfy these requirements, motivating our explicit hierarchy and asymmetric treatment of structure and execution. In particular, basic-motion-level supervision is part of this modeling strategy: it specifies the motion content that the structure representation should preserve. The comparison thus evaluates the proposed modeling strategy as a whole, rather than an architectural change under identical supervision. Within this setting, the results support explicitly assigning representation responsibilities beyond applying reconstruction and semantic losses or introducing a latent-factor split.

Q3: About the closed-bank primitive recovery in longer trajectories.

A3: We sincerely thank you for raising this thoughtful question. Under our hypothesis that reusable motion primitives serve as planning units, the purpose of this experiment is to examine whether our definition of primitive identity supports consistent recovery across different executions and occurrences within longer trajectories. We therefore use a closed-bank setting to evaluate the detection of known reusable structures in longer trajectories.

We use the largest matching registered primitive as the parsing rule, motivated by evidence that grouping actions into macro-actions reduces decision-making costs and accelerates execution [1]. This parsing rule, however, does not guarantee uniquely identifiable boundaries. When an unmatched structure is detected, it is treated as a candidate new primitive for subsequent identification and registration. We will clarify our motivation for this parsing rule and its limitations in the revised manuscript.

[1] A. Dezfouli and B. W. Balleine, “Habits, action sequences and reinforcement learning,” European Journal of Neuroscience, vol. 35, no. 7, pp. 1036–1051, 2012.

Q4: About the description of the training pipeline needs clarification.

A4: We sincerely thank you for pointing out this inconsistency. The support detector was trained with temporal-support annotations, while representation training used saved coefficients extracted from annotated supports and the resulting basic-motion vectors. During raw-trajectory inference, detector-predicted supports were used with the representation models frozen. We will revise Algorithm 1 and the staged-learning description to make this distinction explicit and consistent with Appendix B.1. We conducted an additional matched retraining experiment using annotated versus detector-predicted supports and obtained identical exact structure-recovery rates.

Representation-training supportsSeen-execution structure recoveryHeld-out-execution structure recoveryOrder-only recovery
Annotated100% (468/468)100% (120/120)100% (234/234)
Detector-predicted100% (468/468)100% (120/120)100% (234/234)
Reviewer_Hq6NOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper works to learn reusable motion-primitive representations from executions with a hierarchical model that formalizes the local motions, encode their orders, and estimate the execution factors separately. Experiments on a controlled benchmarks report near-perfect cross-execution retrieval, structure recovery, and close-bank primitive recovery in long trajectories.

Critical strengths

The paper draws a distiction between primitive identity and execution, offering a coherent perspective on the invariances required for primitive reuse.

The proposed system integrates constituent motion localization, structural representation learning, and primitive occurrence recovery into a unified framework.

Critical weaknesses
  1. The main concern is the sythetic benchmark for evaluation. On which, they achieve near-perfect scores (R@1 = 1.000, 100% structure recovery, 1.000 trajectory recovery). No realistic motion dataset is evaluated, and it remains unknown whether the canonical basic-motion decomposition, the restricted transformation family G, or the support-detection assumptions hold in any real setting.

  2. Longer-trajectory recovery assumes a fixed bank of 168 known primitive structures and a supervised constituent-motion detection frontend. The perfect recovery results therefore establish that the model can look up known primitives under clean detection conditions, not that it can discover previously unseen primitives, handle ambiguous compositions, or operate without boundary supervision. These are the conditions that characterize realistic continuous motion, and none of them are evaluated.

Questions for authors

Does the approach generalize to real-world motion data, or to transformation families outside the known execution parameterization? Even a qualitative demonstration on a public motion-capture dataset would help strengthen the paper’s claims.

2 comments

We sincerely thank the reviewer for the careful review and constructive feedback, and for recognizing the coherent perspective offered by our distinction between primitive identity and execution, as well as our unified framework. To provide context for our point-by-point responses, we first clarify the central research question and intended contribution of this work, and then address the reviewer’s concerns point by point. We hope that our clarifications and additional analyses below address the reviewer’s concerns.

Our central hypothesis is that planning for complex action-sequence tasks uses reusable motion primitives as its basic units. The same primitive can be realized through trajectories that vary in execution factors, like duration, amplitude, and relative rhythm. This raises the central question of our work: when should trajectories with different executions be regarded as instances of the same primitive? We therefore represent each primitive instance in terms of structure and execution, while execution captures these realization variations and structure defines primitive identity through the ordered composition of canonical basic motions. The focus of this paper is to make this identity criterion explicit and learn a structural representation that preserves motion geometry and constituent order across execution variations, so that it can serve as a computational interface for using reusable motion primitives in motion planning.

Q1: About the real-world motion data evaluation.

A1: We sincerely thank you for this helpful suggestion. We conducted an additional evaluation on the public HDM05 motion-capture dataset [1], using its normalized 3D skeleton release. We selected forward and backward rotations of the left and right arms and squats as basic motions, and formed primitives containing basic motions from recorded segments. Following the paper, we evaluate cross-execution retrieval, execution exchange, ordered-structure readout, and recovery in longer trajectories, using the same baseline families. The table below reports the results of seed 0 and 1.

MethodRetrieval (R@1) ↑Retrieval (mAP) ↑Execution exchange (nRMSE) ↓Ordered-structure (exact recovery) ↑Long-sequence (joint recall) ↑
OURS1.000 / 1.0001.000 / 1.0000.2202 / 0.22141.000 / 1.0001.000 / 1.000
Sem-AE0.738 / 0.7440.529 / 0.5370.4240 / 0.42120.369 / 0.3750.796 / 0.737
Sem-Factor0.774 / 0.7200.662 / 0.5860.3736 / 0.41290.452 / 0.3270.639 / 0.514

These results support the intended structure representation and execution responsibilities on recorded human motion: our model recognizes the same primitive across different executions, its learned structural representation preserves motion geometry and constituent order, and the execution retains execution-specific variation.

[1] Meinard Müller, Tido Röder, Michael Clausen, Bernhard Eberhardt, Björn Krüger, and Andreas Weber. Documentation: Mocap Database HDM05. Technical Report CG-2007-2, Universität Bonn, 2007.

Q2: About the closed-bank primitive recovery in longer trajectories.

A2: We sincerely thank the reviewer for this insightful comment. We agree that this evaluation addresses closed-bank primitive detection. Under our hypothesis that reusable motion primitives serve as planning units, the purpose of this experiment is to examine whether our definition of primitive identity supports consistent recovery across different executions and occurrences within longer trajectories. The bank provides explicit structural identities against which to test whether execution variations preserve identity and whether different structures remain distinguishable.

The central question here is therefore whether the same registered primitive can be recognized across its varying trajectory realizations, rather than whether previously unseen primitives can be discovered. The longer-trajectory evaluation extends this identity-recovery test from isolated recordings to continuous contexts, within the specified bank and parsing rule.

An unseen primitive would require a mechanism for establishing and registering a new identity. To address the reviewer’s concern, we have added recovery experiments involving unseen actors, unseen canonical local-motion geometries, and unseen constituent orders. The table below reports the recovery results for each setting, averaged over four seeds.

Evaluation settingNumber of test instancesComplete-structure recovery (%)
Unseen actors23484.6
Unseen local geometries23488.5
Unseen orders23492.3

Thank you for you response. The HDM05 results partially address our first concern, and we acknowledge the improvement.

However, the semi-controlled construction of the HDM05 benchmark, the unexplained perfect scores on real data, and the unresolved closed-bank restriction leave our overall assessment unchanged. The authors manually selected specific basic-motion types (arm rotations, squats) and manually segmented recorded motion into primitive instances. This means the canonical basic-motion decomposition and execution family G were still hand-specified for HDM05, not inferred from naturalistic, unscripted motion?

The paper has moved in the right direction but has not yet established the breadth of applicability that justifies acceptance.

Hence i keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.