acceptodds
Under review as a conference paper at ICLR 2027

Second-Order Motion Representations: Grassmannian Differential Kinematics for Unsupervised Hand-Object Interaction Segmentation

Abstract

Representations of human motion are typically built from appearance, spatial configuration, or first-order displacement. We show that what they omit is second-order structure: the rate at which motion direction and speed change. This is not recoverable from zero- or first-order descriptors, yet it is what distinguishes execution modes sharing objects, locations, and appearance - tightening versus loosening, moving versus pouring. We propose a representation that makes this structure explicit. Short windows of 3D scene flow are lifted to points on the Grassmann manifold, yielding a subspace trajectory whose covariant derivatives expose manifold acceleration and direction change as independent, coordinate-free descriptors of how motion evolves. Recovering them from noisy observations poses a representational trade-off: differentiation amplifies noise, while temporal averaging suppresses the transitions that constitute the signal. We resolve this by combining localized Fr\'echet means with a preemptive jump-diffusion process that halts integration across physical discontinuities.We analyse the resulting representation formally, deriving a discrete curvature error bound under stated assumptions and establishing the necessity of second-order features for a constructed class of kinematically ambiguous boundaries. The representation enables causal and training-free boundary detection. Detected segments are subsequently represented by fixed-dimensional spatial path-signature descriptors and grouped by histogram-intersection similarity with complete-linkage clustering, allowing recurring action instances to receive common discovered labels. We test the representation on unsupervised temporal segmentation. When all methods are given the same 3D scene flow, ours performs best, showing that the gains come from the representation and not from the use of 3D input, and it improves over both zero-order models and recent segmentation methods. We validate our error bounds empirically under severe noise and in realistic settings with estimated scene flow, evaluating on OakInk2, HOT3D, and EPIC-KITCHENS. Accuracy degrades predictably as the kinematic signal-to-noise ratio of the input flow falls, showing that our method operates close to the information limit of the input rather than being limited by the representation itself.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Temporal Action Segmentation, Motion Representation Learning, Grassmann Manifolds, Hand--Object Interaction, Unsupervised Action Segmentation.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dctvOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper investigates unsupervised temporal segmentation of hand-object interactions using second-order motion features. The authors propose a training-free geometric framework that maps 3D scene flow onto Grassmannian manifolds and detects action boundaries through manifold acceleration and direction changes. Fréchet smoothing and jump-diffusion improve noise robustness, while spatial signatures and hierarchical clustering group similar motion segments. The method is evaluated on OakInk2, HOT3D, and EPIC-KITCHENS, showing improvements over zero-order representations and other baselines. Additional experiments examine noise robustness, scene-flow quality, and component ablations.

Critical strengths
  1. The paper addresses unsupervised fine-grained HOI segmentation, an important problem in motion understanding. Exploring second-order motion features to distinguish subtle action transitions is a promising direction.
  2. The paper introduces a training-free framework using Grassmannian geometry, manifold acceleration, and direction-change features. Results across multiple datasets show improvements over zero-order and other baselines, suggesting the potential of this approach.
Critical weaknesses
  1. Table 1 shows limited improvements over the zero-order Grassmannian baseline, particularly on HOT3D. The comparisons suggest that much of the performance gain may come from the Grassmannian representation itself, rather than the proposed second-order features. Given that second-order motion representation is the core contribution, more targeted experiments are needed to demonstrate its necessity for distinguishing fine-grained motion transitions.
  2. Although the supplementary qualitative results show meaningful action boundaries, the role and practical value of the clustering stage remain unclear. The method assigns cluster IDs to motion segments, but its contribution to HOI understanding is not sufficiently demonstrated. Beyond ARI and NMI, evaluations of semantic interpretability or downstream tasks, such as action retrieval, procedural analysis, and annotation assistance, would strengthen the work. Additionally, reducing redundant descriptions in the main paper and including more intuitive figures and stronger experimental results would help better support the paper's central claims.
Questions for authors
  1. Grassmannian subspaces may discard motion direction and magnitude information, as (v) and (-v) span the same subspace. How does the proposed representation distinguish motion reversals such as tightening versus loosening? Can authors provide controlled experiments addressing this potential information loss?
  2. Algorithm 2 and Appendix A.2 use different temporal normalizations for manifold acceleration. Could authors clarify the relationship between the implemented discrete feature and continuous covariant acceleration, and provide a rigorous justification for the error bound in Theorem 3.1?
  3. Although Table 1 and Appendix Table 7 provide relevant comparisons, it remains unclear whether the improvements can be attributed specifically to second-order features, as some baselines use different detection procedures. Could authors clarify whether the zero-order baseline shares all other settings and provide more targeted evaluations of fine-grained motion transitions?
Reviewer_AB4kOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work presents a geometric framework for unsupervised HOI segmentation using Grassmannian differential kinematics. The experimental results on the OakInk2 and HOT3D datasets demonstrate competitive performance.

Critical strengths

The experimental results on the OakInk2 and HOT3D datasets demonstrate competitive performance. The EPIC-KITCHENS evaluation show algorithm robustness on cluttered environment. The video in the supplementary material is clear and helpful for paper understanding.

Critical weaknesses
  1. The Introduction section is too lengthy, and the key motivation is not clearly presented.

  2. The paper lacks an overall framework diagram of the proposed method, which diminishes its readability.

  3. The pseudocode takes up too much space.

  4. The paper provides only Figure 1 as an example. No failure cases are presented or discussed.

  5. The proposed approach heavily depends on the quality of the upstream scene flow. The paper does not quantify how much of the performance gap between the proposed method and the learned baselines is attributable to flow quality rather than representation design.

Questions for authors
  1. The authors should decompose the performance gap into flow-quality vs. representation-design components, and show robustness with flow estimation error.

  2. The authors should provide CPU/GPU runtime benchmarks to support the "CPU-deployable" claim.

Reviewer_YZD3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new motion representation for unsupervised hand-object interaction segmentation from 3D point cloud sequences. The method lifts 3D scene flow to trajectories on the Grassmann manifold, smooths them with Fréchet means, and computes second-order tangent-space features (manifold acceleration and direction-change angle) via parallel transport, combined with a jump-diffusion reset at abrupt transitions. Segment boundaries are detected from the resulting kinematic scores, and the detected segments are then grouped by clustering. The paper also provides a theoretical analysis, including an error bound on the discrete curvature and a constructed example showing that second-order features can distinguish trajectories that zero-order distances cannot. Experiments show promising results.

Critical strengths
  1. The motivation is clear, and the overall method design makes sense.
  2. The mathematical derivations are generally well-organized and rigorous.
Critical weaknesses
  1. Writing
    • Section 1 mixes the introduction with related work. The core motivation is simple: zero- and first-order motion features are insufficient, and second-order features, capturing changes in both magnitude and direction, are needed. While this idea is easy to understand, the section restates it repeatedly, which slows the pace and lowers the information density. The overall tone relies heavily on non-academic and rhetorical wording, reading more like a blog post than a professional academic paper.
    • Many citations throughout the paper should use \citep instead of \citet. Otherwise, many sentences are hard to read, and the flow of the text is broken.
    • Algorithms 1 and 2: for CS papers, “Input” and “Output” are preferable to “Require” and “Ensure”.
    • The terms “zero-order” and “first-order” are used inconsistently and are never formally defined. Technically, I would expect zero-order to refer to position, and first-order to velocity or displacement within a fixed time interval. However, the paper uses these two terms interchangeably, which causes confusion.
  2. Evaluation
    • The main paper reports only one quantitative comparison (Table 1), which is rather thin; most additional results are deferred to the appendix. Moreover, the evaluation labels for OakInk2 and HOT3D were re-annotated by the authors’ team (Lines 385–387), so the comparison is not entirely neutral. The baselines are also not designed for the same input. Although the paper includes some controlled comparisons, the reported baseline results may not reflect the best achievable performance of those baselines.
    • The proposed pipeline has many components, yet the main paper contains only one ablation (zero-order vs. second-order features in Table 1) and a single qualitative example (Fig. 1). All other ablations and analyses are deferred to the appendix. Therefore, the main paper does not convey what role each component plays. Moreover, the paper relies heavily on theoretical derivations and provides little visualization or analysis to help build intuition.
    • This representation seems applicable to many related motion tasks. Why is it evaluated only on hand-object interaction segmentation? Evaluating on additional related tasks would make the results more compelling.
  3. Incorporating second-order information into motion representations is not a completely new idea. The paper should discuss and compare with prior methods that use second-order information to clarify its distinct contribution. For example, Rao et al. (2002) detect action boundaries from spatiotemporal curvature of trajectories, and this work is not cited.

Rao, Cen, Alper Yilmaz, and Mubarak Shah. "View-invariant representation and recognition of actions." International journal of computer vision 50.2 (2002): 203-226.

Questions for authors
  1. Why is Qwen3.6-27B chosen as a baseline? This choice is quite unusual. For a paper in the traditional CV style, I would expect comparisons with other traditional methods, possibly complemented by some deep learning methods.
    • Even if the goal is to compare against a recent, strong VLM, a frontier model such as GPT would typically be chosen, as it is both stronger and more representative.
    • The main advantage of Qwen3.6-27B is that it is open-source and thus easy to fine-tune, but the paper evidently does not fine-tune the VLM, so choosing Qwen3.6-27B offers little benefit.
    • VLMs are designed for generality. Outperforming a VLM on a highly specialized task does not demonstrate much, so this comparison has limited significance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.