SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding

Human object interaction (HOI), gaze pattern, and their anticipation are intricately linked, providing valuable insights into cognitive processes, intentions, and behavior. However, most existing models handle gaze and actions separately, missing both their interdependence and the advantages of a unified solution. This paper presents a novel unified framework, SAGE (Synchronized Action-GazE), which integrates simultaneous recognition and anticipation of both HOI and human gaze into a single unified end-to-end trainable model. Our approach leverages a transformer-based architecture and incorporates gaze data into spatiotemporal attention mechanisms to simultaneously predict current and future human actions and gaze behavior. We explore this bidirectional relationship between gaze and actions under different scenarios, whether requiring a close-up, detailed view (egocentric) or a wider, more contextual view (exocentric), making our framework versatile for various applications. Additionally, due to lack of datasets for comprehensive analysis of both HOI and gaze in exocentric videos, we establish a new benchmark Exo-Cook using the Ego-Exo4D dataset to facilitate further research in this domain. Experiments on three benchmark datasets—VidHOI, EGTEA Gaze+, and Exo-Cook—show that jointly modeling gaze and actions across current and future frames achieves consistently strong results, often surpassing specialized state-of-the-art models tailored to individual tasks. Our experiments on three benchmark datasets: VidHOI, EGTEA Gaze+ and Exo-Cook, demonstrate that the synergy between gaze and actions in the current and future frames compares favorably and even outperforms individual task specialized state-of-the-art models. By unifying actions and attention in a comprehensive way, our work lays the groundwork for more intuitive human-machine interaction.