- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a dataset, EgoIntrospect, which provides synchronized internal states, such as physiological signals, recording beyond conventional audio, video, and human motion recording on egocentric wearable assistants such as AI glasses. The dataset provides large-scale recordings across extensive subjects with user self-annotations. The paper proposes to use internal states to enable personal assistant applications on three newly designed tasks.
Critical strengths
-
The authors synchronize multiple wearable devices and collect extensive participant annotations. The collection and processing procedures are carefully documented, and the effort required to obtain this combination of data is considerable.
-
Combining physiological signals with egocentric observations and wearer-reported experiences is promising for passive assessment of internal states. It may eventually support more proactive assistance, although such applications also require greater reliability due to the potential consequences of inappropriate interventions. Relative to existing egocentric datasets, the strongest motivation for this collection is the combination of physiological sensing and wearer-provided subjective annotations.
Critical weaknesses
-
The paper highlights physiological signals as new modalities compared with previous datasets. Yet the ablations on physiological signals on the state-of-the-art MLLMs do not seem promising, and fine-tuning state-of-the-art models is underexplored (see Question 2(c)). It is unclear whether adding physiological signals is necessary and helpful.
-
The paper’s organization is unclear, and its contribution beyond the dataset is moderate. All the benchmark and key findings are in appendices, which are difficult to find. Despite the details in data collection and task definitions, the insights beyond collecting new modalities remain unclear. What are the gaps in enabling MLLMs to understand internal states and physiological signals? What are the promising directions for addressing these problems, given there is already existing research on physiological signal supported MLLM like SensorLM [1]?
-
Reproducibility issue. The key contribution of the paper is the dataset and benchmark itself yet it is not accessible to the reviewer. The reviewer therefore asks the authors to share a small subset or demo of the recorded data for review.
Questions for authors
Methods: The in-situ annotation protocol requires participants to intentionally say commands such as “Hey assistant…”. Could speaking these commands influence the internal states being recorded, compared with a condition without verbal commands? Have the authors conducted any comparison to assess this potential bias?
Benchmark and Evaluation: Previous research such as SensorLM [1] explores sensor-language modeling using physiological features, including PPG-derived features. Yet this paper does not consider it as a potential baseline in the benchmark. Gaze-related modeling with VLMs has also been explored in previous research, such as [2], [3], and [5]. MemoLens [2], for example, fine-tunes a VLM for gaze understanding. This paper does not consider these methods as baselines or discuss how they could be adapted to its tasks.
Unlike previous dataset papers [4] and [5], this paper does not fine-tune a model to investigate whether physiological signals can improve MLLMs’ understanding, yet directly integrating physiological signals in the state-of-the-art MLLMs has marginal improvements. The only direct physiology ablation is in Table 16, which compares video + gaze + transcript with video + gaze + transcript + physiology on the EI tasks. The improvement seems marginal or even negative. The absolute accuracy on MCQs is below 60%.
Given that real-world applications often require open-ended responses, these results raise questions about practical performance. For scenarios such as proactively offering help finding a museum’s location in Fig. 4, what additional benefit do physiological signals provide beyond video, audio, and gaze? How does this assistance compare with a straightforward voice interface through which users explicitly request the location?
Bibliography [1] Zhang, Yuwei, et al. "SensorLM: Learning the language of wearable sensors." Advances in Neural Information Processing Systems 38 (2026): 46813-46849. [2] Alam, Samiul, Shakhrul Iman Siam, and Mi Zhang. "MemoLens: Empowering Augmented Reality Glasses with Super Memory." Proceedings of the 24th Annual International Conference on Mobile Systems, Applications and Services. 2026. [3] Mathew, Athul M., et al. "Gazevlm: A vision-language model for multi-task gaze understanding." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026. [4] Yang, Jingkang, et al. "Egolife: Towards egocentric life assistant." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025. [5] Huang, Yifei, et al. "Egoexolearn: A dataset for bridging asynchronous ego-and exo-centric view of procedural activities in real world." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.