acceptodds
Withdrawn from ICLR 2027

EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite extensive efforts on egocentric video datasets and benchmarks, understanding users' internal states, crucial for enabling seamless AI assistant experiences, remains largely overlooked. In this work, we introduce EgoIntrospect, the first egocentric dataset captured in user-driven scenarios with self-annotations that explicitly reveal users' interactive intentions with AI assistants. EgoIntrospect was collected using a cross-device setup, providing synchronized video, audio, gaze, motion, physiological signals. It consists of 180 hours of recordings from 60 subjects, with an average clip duration of 3h. Leveraging EgoIntrospect, we formalize a suite of tasks centered on user internal states, including affective experience, interactive intent, and cognitive memory. We further process the annotations to construct benchmarks that evaluate the ability of modern multimodal large language models (MLLMs) to reason about users’ internal states from egocentric observations. Experiments on our benchmark suggest that existing MLLMs struggle to effectively leverage multimodal signals to infer users’ subjective internal states. The dataset and annotations will be made publicly available to advance research in egocentric vision and wearable AI assistants.

Primary area:
datasets and benchmarks
Keywords:
Ego-centric, smart glasses, user intention, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XpH4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a dataset, EgoIntrospect, which provides synchronized internal states, such as physiological signals, recording beyond conventional audio, video, and human motion recording on egocentric wearable assistants such as AI glasses. The dataset provides large-scale recordings across extensive subjects with user self-annotations. The paper proposes to use internal states to enable personal assistant applications on three newly designed tasks.

Critical strengths
  • The authors synchronize multiple wearable devices and collect extensive participant annotations. The collection and processing procedures are carefully documented, and the effort required to obtain this combination of data is considerable.

  • Combining physiological signals with egocentric observations and wearer-reported experiences is promising for passive assessment of internal states. It may eventually support more proactive assistance, although such applications also require greater reliability due to the potential consequences of inappropriate interventions. Relative to existing egocentric datasets, the strongest motivation for this collection is the combination of physiological sensing and wearer-provided subjective annotations.

Critical weaknesses
  • The paper highlights physiological signals as new modalities compared with previous datasets. Yet the ablations on physiological signals on the state-of-the-art MLLMs do not seem promising, and fine-tuning state-of-the-art models is underexplored (see Question 2(c)). It is unclear whether adding physiological signals is necessary and helpful.

  • The paper’s organization is unclear, and its contribution beyond the dataset is moderate. All the benchmark and key findings are in appendices, which are difficult to find. Despite the details in data collection and task definitions, the insights beyond collecting new modalities remain unclear. What are the gaps in enabling MLLMs to understand internal states and physiological signals? What are the promising directions for addressing these problems, given there is already existing research on physiological signal supported MLLM like SensorLM [1]?

  • Reproducibility issue. The key contribution of the paper is the dataset and benchmark itself yet it is not accessible to the reviewer. The reviewer therefore asks the authors to share a small subset or demo of the recorded data for review.

Questions for authors

Methods: The in-situ annotation protocol requires participants to intentionally say commands such as “Hey assistant…”. Could speaking these commands influence the internal states being recorded, compared with a condition without verbal commands? Have the authors conducted any comparison to assess this potential bias?

Benchmark and Evaluation: Previous research such as SensorLM [1] explores sensor-language modeling using physiological features, including PPG-derived features. Yet this paper does not consider it as a potential baseline in the benchmark. Gaze-related modeling with VLMs has also been explored in previous research, such as [2], [3], and [5]. MemoLens [2], for example, fine-tunes a VLM for gaze understanding. This paper does not consider these methods as baselines or discuss how they could be adapted to its tasks.

Unlike previous dataset papers [4] and [5], this paper does not fine-tune a model to investigate whether physiological signals can improve MLLMs’ understanding, yet directly integrating physiological signals in the state-of-the-art MLLMs has marginal improvements. The only direct physiology ablation is in Table 16, which compares video + gaze + transcript with video + gaze + transcript + physiology on the EI tasks. The improvement seems marginal or even negative. The absolute accuracy on MCQs is below 60%.

Given that real-world applications often require open-ended responses, these results raise questions about practical performance. For scenarios such as proactively offering help finding a museum’s location in Fig. 4, what additional benefit do physiological signals provide beyond video, audio, and gaze? How does this assistance compare with a straightforward voice interface through which users explicitly request the location?

Bibliography [1] Zhang, Yuwei, et al. "SensorLM: Learning the language of wearable sensors." Advances in Neural Information Processing Systems 38 (2026): 46813-46849. [2] Alam, Samiul, Shakhrul Iman Siam, and Mi Zhang. "MemoLens: Empowering Augmented Reality Glasses with Super Memory." Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services. 2026. [3] Mathew, Athul M., et al. "Gazevlm: A vision-language model for multi-task gaze understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [4] Yang, Jingkang, et al. "Egolife: Towards egocentric life assistant." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [5] Huang, Yifei, et al. "Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Reviewer_z5j1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a 180-hour, 60-participant egocentric dataset (video, gaze, audio, physiology from a wristband and ring) recorded during unconstrained 2-4 h daily sessions, with participants annotating their own affect, requests, interruptibility, and memory. A 4,060-item benchmark over 11 sub-tasks is built from these annotations and 21 VLMs are evaluated.

Critical strengths
  1. The data design is novel and costly to replicate: self-annotated internal states on naturalistic long recordings with synchronized physiology and gaze.
  2. A test-retest study and human baselines is used, which most dataset papers omit.
Critical weaknesses
  1. The data and labels do not reflect the paper's claims. Physiology, the advertised differentiator, is unvalidated and confounded. The introduction motivates it as a “more direct” measurement of internal processes, but the paper gives no evidence that the collected EDA/PPG/HR carry any affect signal (no simple classifier, no correlation with self-report). Recordings include sports, cycling and hiking, and given that EDA/PPG/HR track exertion and motion, no discussion about artifact removal or exertion control is described on how activities could confound the results. Sampling rates for the wristband and ring, and any alignment-drift check over 2-4 h, are missing. The one physiology ablation shows no benefit (under one point), and the signal’s input format to the VLM is never stated. Some targets are ill-posed without user history since ground truth is one person’s private state and the model (and the human baseline raters) get no history of that person beyond a one-shot ICL example. On re-test, participants themselves reproduce the emotion-intensity and accepted-recommendation labels only ~56% of the time, roughly where the best models already perform. The benchmark cannot separate “models fail” from “the target is unpredictable from the given inputs.”

  2. The reported evidence does not support the claims and cannot be checked. The main manuscript abrupbtly ends, with no results section or discussion present. The appendix results are small-sample (15 test participants, several tasks with 88-200 items), have no confidence intervals, and understate trivial baselines (recording-test majority class is 66%, the “top” model scores 68%). The paper claims to be the “first multimodal egocentric dataset and benchmark designed for understanding user internal states in a naturalistic, daily-life setting” when existing literature already covers that claim.

Questions for authors
  1. Report majority-class baselines and participant-clustered CIs for all tables. Which gaps survive?
  2. How would a video-blind, text-only baseline on the Gemini-generated distractor tasks perform?
  3. How are physiological signals input to the models?
  4. What is the common sampling rate used for alignment between the multimodal signals?
  5. What is the residual alignment error/drift per stream over 2-4 h.
Reviewer_sfNpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces EgoIntrospect: 60 participants each record a 2-4 hour self-chosen daily-life session with Pupil Labs Neon glasses, an Apple Watch, a wristband and a smart ring. Labels come from the participants themselves, through in-situ voice commands and retrospective annotation. The derived EgoIntrospectQA benchmark (4,060 instances, 15-participant test split) covers six tasks across affective experience, interactive intent and cognitive memory. Validation consists of zero-shot evaluation of 21 open and closed MLLMs, small human-predictor subsets, input-modality and in-context ablations, and a 5-participant six-month test-retest study. All results are in the appendices.

Critical strengths
  • The labels come from the wearer (in-situ and retrospective), not from external annotators. Internal states such as intent and memorability cannot be labeled by third parties, so this is the dataset's core differentiator from Ego4D-style datasets.
  • The dataset has many synchronized modalities (video, audio, gaze, IMU, physiological signals) captured in naturalistic, subject-chosen settings across 14 scenario types. Few public egocentric datasets pair in-the-wild video with physiology, so it could be useful beyond this benchmark.
Critical weaknesses
  • The annotation quality is questionable. Test-retest uses only 5 participants, relies on an LLM judge and a lenient box criterion, and shows exact agreement of only 55.6% for emotion intensity and 56.5% for accepted recommendations, although tasks are built on these labels. Parts of the ground truth are constructed rather than self-annotated: distractors are generated by Gemini and the correct option is 'subtly adjusted' to match them, some timestamps are set by experimenters, and some options are rewritten by an LLM. This risks LLM-style artifacts and bias for or against Gemini-family models. Human performance is measured on subsets of about 10% (as few as 5-29 items) with no description of who the predictors were, and the gaps are small (emotion intensity best model 55.7 vs human 60.0; memory lifespan 85.4 vs 88.9 with a 74% majority class). No confidence intervals or significance tests are given, so the claim that MLLMs struggle may partly reflect label noise.
  • The main paper contains no results. All numbers, baselines, human scores and ablations are in Appendices G-N, so the abstract's claim is unsupported within the paper body. The physiological signals, a headline modality, are used in only one task and add about 0 points (56.6 to 56.2 for Kimi), and how they are serialized for MLLMs is not described.
Questions for authors
  1. How were the distractors generated and validated for the tasks that use them (emotion with reason, request prediction, memory recall, memory-answer tasks), and how was the correct option adjusted to match them? Did you check that the correct answer is not identifiable from style alone?
  2. Which input condition does Table 15 actually report (it matches the video-only rows of Table 16), and how exactly are the physiological signals given to each MLLM?
  3. Who were the human predictors (participants, authors, crowd), how many per task, and what are confidence intervals for both human and model scores?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.