EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning

Despite extensive efforts on egocentric video datasets and benchmarks, understanding users' internal states, crucial for enabling seamless AI assistant experiences, remains largely overlooked. In this work, we introduce EgoIntrospect, the first egocentric dataset captured in user-driven scenarios with self-annotations that explicitly reveal users' interactive intentions with AI assistants. EgoIntrospect was collected using a cross-device setup, providing synchronized video, audio, gaze, motion, physiological signals. It consists of 180 hours of recordings from 60 subjects, with an average clip duration of 3h. Leveraging EgoIntrospect, we formalize a suite of tasks centered on user internal states, including affective experience, interactive intent, and cognitive memory. We further process the annotations to construct benchmarks that evaluate the ability of modern multimodal large language models (MLLMs) to reason about users’ internal states from egocentric observations. Experiments on our benchmark suggest that existing MLLMs struggle to effectively leverage multimodal signals to infer users’ subjective internal states. The dataset and annotations will be made publicly available to advance research in egocentric vision and wearable AI assistants.