TraceForcing: Learning from Persistent Interaction Traces for Fine-Grained Hand-Object Interaction Video Understanding

Fine-grained hand-object interaction (HOI) understanding is not merely about recognizing hands, objects, and actions in individual frames. It requires preserving a persistent interaction trace: who interacts with what, where contact occurs, and how these relations evolve into manipulation processes and state changes over time. However, existing approaches typically model entities, trajectories, or semantic relations as isolated structured cues, lacking a unified mechanism to preserve localized contact evidence and persistent hand-object identities throughout videos. In this work, we propose TraceForce-HOI, a unified framework that models HOI videos through Persistent Interaction Traces, a structured representation capturing hand entities, object instances, contact evidence, and interaction state evolution. To construct reliable traces, we introduce Fast-Slow Trace Identification, which combines dense interaction-aware perception with sparse temporal propagation to capture short-lived contact dynamics while maintaining consistent instance identities over time. The resulting traces are encoded in both pixel and latent spaces through HOI mask videos and instance-aware trace marks, exposing entity identities, active-object states, and localized interaction evidence to the model. Furthermore, we introduce causal identity and contact-attribution constraints to regularize intermediate visual representations, ensuring that interaction evidence remains anchored to the correct entities throughout the video. Extensive experiments demonstrate that TraceForce-HOI achieves significant improvements in fine-grained HOI video understanding, particularly on tasks requiring persistent instance association and interaction evolution modeling, including procedural understanding, state transition recognition, and object part-level comprehension.