acceptodds
Withdrawn from ICLR 2027

What a Choreography Fingerprint Must Forget: Learning Motion Tokens Beyond Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Motion tokenizers trained to reconstruct human movement are widely used as general-purpose motion representations. We study whether their tokens identify a choreography when a different person performs it, and introduce the task of Choreography Fingerprinting: given a short and possibly occluded fragment of a performance, a system must find other performances of the same choreography in an index that grows without retraining, and reject choreographies it does not contain. The task matters in practice: a viral choreography is repeated by thousands of people on social media, and courts have started to consider whether a sequence of movements was copied from another work. A fingerprint for this task has to ignore the performer, the camera and unseen parts while keeping the order of movements, and reconstruction requires neither. We show that codes can be invariant or have high entropy and still carry no information about the choreography. Following how dance notation records a choreography, as ordered movements of body parts with the floor path kept apart, we propose ChoreoMatch: trained on choreography labels without a reconstruction target, it infers hidden body parts without treating them as evidence and stores each third of a second in a few bytes. To test identification beyond the studio, we introduce the Choreography Fingerprinting Dataset of skeleton sequences extracted from amateur videos of viral choreographies filmed on phones, annotated by choreography. Our framework identifies choreographies in these videos and outperforms reconstruction tokenizers, suggesting that a choreography is recognized by what a representation is trained to keep rather than by what it can reconstruct.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
fingerprinting, motion representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YvB9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ChoreoMatch, a choreography fingerprinting pipeline for identifying the same dance routine across different performers and cameras. The core idea is to train compact motion tokens for choreography identity rather than reconstruction, so the representation forgets performer- and camera-specific cues while preserving movement order. Experiments on studio data and a new phone-video choreography dataset show that reconstruction-based motion tokenizers are poorly performe on this task, while the proposed fingerprint retrieves matching choreographies more reliably.

Critical strengths
  1. The problem formulation is sharp and practically motivated. The paper makes a clean and important distinction: a representation that reconstructs human motion well is not necessarily a representation that identifies a choreography. That separation is useful beyond this specific system.
  2. Studio motion capture can hide the performer, camera, and occlusion factors that fingerprinting must survive in the wild, and the new dataset makes the inadequacy of reconstruction tokenizers more convincing.
Critical weaknesses
  1. The method introduces choreography labels into contrastive training, which adds supervision beyond reconstruction or pretrained motion objectives. Architecture and training recipe change together under this signal, but the experiments do not isolate the two factors. The reported gains therefore leave open a basic confound: how much comes from the proposed fingerprint network, and how much would appear if existing motion encoders were trained with the same choreography-contrastive objective alone.
  2. The phone-video dataset pipeline is underspecified. Monocular SMPL estimation from in-the-wild video remains an unsolved problem, yet the paper only says that social-media clips are converted into skeleton sequences, without naming the estimator, reporting its accuracy. Without that pipeline, it is hard to judge how much fingerprinting depends on pose quality. The paper also does not show how the method behaves when parts of the estimated SMPL are occluded or systematically inaccurate under the same phone-video regime.
Questions for authors
  1. If existing baselines are kept fixed and only retrained with the same choreography-contrastive objective, without the proposed architecture, how do they compare with ChoreoMatch?
  2. Which method was used to estimate SMPL on the phone-video set, and what is its accuracy? Relative to the synthetic masks in Appendix B.1, how does retrieval change on phone videos when body parts are truly missing or the corresponding SMPL estimates are inaccurate?
Reviewer_TLeDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces choreography fingerprinting task, which is about identifying a choreography from a short fragment performed by a different person under different recording conditions, and rejecting choreographies that are not in the index. The authors first argue with a preliminary experiment that reconstruction-based motion tokens do not carry choreography identity. They then propose ChoreoMatch, an encoder trained only with choreography labels and masked attention so that hidden joints are never used as evidence. The paper also introduces a dataset of skeletons extracted from amateur phone videos.

Critical strengths
  • The task is well motivated and clearly defined. Choreography identification is emerging tasks, but has received little attention than music copyrights.
  • The paper explains clearly why reconstruction tokens fail for this task supported by experiments
Critical weaknesses
  • The dataset and the experimental setup are unclear.
    • The proposed dataset is not described. There are no statistics, no description of the pose estimation pipeline, and no details on annotation.
    • The meaning of a choreography label is not defined. Table 1 lists "routine" for AIST++ and "trend" for the proposed dataset, but it is unclear how a choreography is designed, at what granularity labels are assigned, and how variants of the same trend are handled.
    • The splits are not specified.
  • There are no qualitative results. The paper does not show retrieved matches, failure cases or examples from the in-the-wild dataset.
  • The baseline comparison is not fair. ChoreoMatch is trained with choreography labels, while most baselines are trained without them. The baselines also use different matching algorithms (token histograms, mean pooling, DTW), so Table 2 mixes the effects of the training signal, the encoder, and the matching algorithm.
Questions for authors
  • Figure 7 and 8 are in the wrong order.
  • How is the observation mask MM determined for the Choreography Fingerprinting Dataset? Is it based on 2D keypoint confidence? How does performance change under intermittent occlusion, or when hidden joints are filled by the pose estimator with high confidence?
  • How robust is ChoreoMatch to temporal misalignment, such as different tempo or delayed beats, and to mirrored performances? Both are common in amateur videos. The ordered anchor concatenates frames and joints in a fixed order.
  • How are the baselines trained? Are the reconstruction tokenizer and MoMask trained or fine-tuned on AIST++ and the proposed dataset, or used with weights from pretrained model?
  • To separate the contribution of the encoder from the retrieval engine, could you report baseline features scored with the CRE, and ChoreoMatch descriptors scored with DTW, on the proposed dataset?
  • Are the AIST++ query choreographies disjoint from those used for training?
Reviewer_4yntOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ChoreoMatch for choreography fingerprinting. A spatio-temporal encoder-only model is introduced for learning choreography identity, which is trained under a supervised contrastive learning paradigm and further enhanced by mask-aware processing for partial observations. In addition, a choreography retrieval engine is designed for final fingerprint matching using cosine similarity between descriptors reconstructed from residual vector quantization codes, together with temporal offset alignment. Experimental results on AIST++ and a newly introduced dataset demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The proposed method is technically sound. The authors introduce a spatio-temporal encoder-only model and train this model by supervised contrastive learning, which can effectively achieve choreography fingerprint encoding

  2. The ablation studies in the paper effectively demonstrate the factors that influence choreography fingerprint encoding.

Critical weaknesses
  1. The motivation of the paper is not sufficiently justified. Much of the discussion focuses on the limitations of reconstruction-trained motion tokenizers, while the proposed method learns representations through supervised contrastive learning. The connection between these limitations and the specific proposed design is not clearly established.

  2. The comparison experiment in table 2 appears somewhat insufficient. For example, the proposed baseline has relatively low performance, and the cited previous works are somewhat dated and were not specifically trained on this task.

Questions for authors
  1. Could the authors include representative recent reconstruction-based models as baselines, such as motion tokenizers or VAEs used in large motion generation models?

  2. It would be helpful if the authors could provide some visualizations of the matching results.

  3. Hope the authors could further clarify the motivation of this work

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.