acceptodds
Under review as a conference paper at ICLR 2027

StimEM: Stimulus-Conditioned Masked Modeling for Eye-Movement Representation Learning

Abstract

Eye movements provide rich temporal information about human behavior and cognitive function, offering a precise, non-invasive means of assessing cognitive and neurological disorders. However, existing approaches typically rely on handcrafted metrics or task-specific supervised models, whose performance and generalizability are constrained by limited training data and model capacity. Inspired by the success of large language models and foundation models across diverse modalities, we introduce StimEM, an eye-movement foundation model based on Stimulus-Conditioned Eye Movement Masked Modeling. StimEM first learns an ocular tokenizer using finite scalar quantization, jointly optimizing waveform reconstruction and the prediction of informative behavioral features, preserving both local signal structure and trial-level information relevant to cognition. We then pretrain a novel stimulus-conditioned transformer to predict masked tokens. Its structured attention architecture captures dependencies within ocular responses and their relationships with external stimuli. A multi-scale masking strategy further encourages learning of local dynamics and longer-range temporal structure. Pretrained on approximately 500 hours of stimulus-driven eye-tracking recordings spanning multiple cognitive tasks, StimEM achieves state-of-the-art performance on six downstream tasks related to cognitive assessment, demonstrating strong representation learning capabilities and generalizability. Our work first demonstrates the value of self-supervised pretraining for eye-movement data and provides a foundation for further research and broader applications in cognitive science and healthcare.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Eye movement, foundation model, self-supervised learning, representation learning, cognitive assessment

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_b8M6OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StimEM, a stimulus-conditioned foundation model for learning the representation of eye-tracking signals. The method first jointly trains an FSQ-based ocular tokenizer through waveform reconstruction and trial-level behavioral feature prediction. It then freezes the tokenizer, uses the discrete code as the masked modeling target, and pre-trains a stimulus-conditioned Transformer. The paper uses approximately 4,255 participants and 500 hours of eye-tracking data for pretraining and evaluates the model on six clinical classification tasks. Additionally, the authors conducted extensive ablation experiments to validate the effectiveness of the method.

Critical strengths
  1. In the field of eye-tracking modeling, a pretraining corpus comprising approximately 500 hours of data and 4,255 participants is quite substantial. This in itself represents a significant contribution to the field.

  2. The authors conducted experiments across multiple tasks, as well as thorough ablation studies, to validate the effectiveness of the method described in this paper.

Critical weaknesses
  1. In Sec. 4.1, the authors explicitly state that the pretraining split and the downstream split are constructed independently. Appendix A.3 directly refers to the current protocol as the “transductive classification protocol.” In other words, although the downstream train, validation, and test sets are subject-disjoint internally, there is no guarantee that the downstream test subjects are excluded from the tokenizer training and masked pretraining. I would like the authors to first specify the exact number or proportion of test subjects from the six downstream tasks that were seen during pretraining.

  2. In Section 4.4, the authors analyze data scaling using pretraining data fractions of 12.5%, 25%, 50%, and 100%. However, Appendix C.1 explicitly states that different configurations share the same frozen 12-layer FSQ tokenizer. Therefore, I would like the authors to clarify: How much pretraining data was used to train this tokenizer itself? If it was trained on 100% of the corpus, then even if the masked pretraining encoder only sees 12.5% of the data, its targets still come from a teacher/tokenizer that has already seen 100% of the data.

Since this is not my area of expertise, I hope AC will take my comments into consideration as appropriate.

Questions for authors

see weaknesses.

1 comment

Since I'm not familiar with this topic, I will maintain my rating score.

Reviewer_4HtDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes StimEM, a framework for learning eye-movement representations from synchronized visual stimuli and binocular recordings. The authors are trying to answer some very timely questions that 1 whether the gaze signal can be tokenizer, 2 the stimulus conditional based eye movements to construct computational model to link ocular responses and their relationships with external stimuli.

Critical strengths

Strengths 1, The method is formulated, directed attention mask maintains stimulus-only representations, and ocular tokens can access stimulus and response context. 2, The tokenizer combines complementary training objectives. Waveform reconstruction and behavioral-feature prediction provide a reasonable way to preserve both temporal signal details and task-relevant response descriptors, 3, Aggregating trials within each acquisition task and then across tasks matches the clinical label’s unit of analysis and prevents tasks with more trials from dominating the prediction. 4, 500 hours dataset is a hugh work, which should be acknowledged. If open-source for sure.

Critical weaknesses

Weakness: 1, The authors did not evaluate their methods on other stimulus based eye movement datasets, like GazeBase, GazeBaseVR, Judo 1000, etc. The used dataset is good, but not release yet, so 2, The evaluation does not establish the claimed generalization to unseen patients. A.3 explicitly describes a transductive protocol, and the supplementary configurations allow unlabeled target recordings during pretraining, so downstream subject-disjoint splits do not ensure that test participants were unseen during representation learning 3, The research gap is insufficiently established against closely related literature. ORASIS-MAE (2024), DiffSynEye, EKYT, etc. 4, StimEM applies a nonlinear classifier to each trial and averages logits, whereas the baselines average trial features before classification; which is unfaired. 5, Task instructions/prompts/stimulus explainations are missing from the encoder despite their importance to the motivation. Authors need to explose more info. 6, The clinical interpretation remains vulnerable to cohort confounding.

Questions for authors

1, How many downstream test participants were included in tokenizer or encoder pretraining, and are results available with these participants excluded from both stages? 2, Does StimEM retain its advantage when the baselines use the same classifier and trial-aggregation scheme? 3, What is the specific methodological advance over ORASIS-MAE, and how does StimEM relate to EKYT and DiffEyeSyn? 4, Why were datasets such as GazeBase, GazeBaseVR, and JuDo1000 not used to assess representation transfer, and what data will be released for independent verification? 5, How were demographic, medication, and recording-site differences between clinical groups assessed or controlled?

Reviewer_kUUzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In their paper, the authors introduce StimEM, a Transformer-based foundation model for eye-movement based subject classification. StimEM integrates binocular with stimulus information. The model training consists of three parts where first an ocular tokenizer is trained by enforcing both waveform reconstruction and trial-level behavioral feature prediction. Further, the model is pretrained to through predicted of ocular tokens from stimulus and eye trial data masked across different time spans. Finally, the model is fine-tuned for a specific test application. StimEM is compared against SOTA general time-series classifier models and achieves superior classification performance on eye movement based classification of six patient-control datasets. Further, ablations in terms of pretraining data size, tokenizer loss, attention mechanism, and masking strategy are provided.

Critical strengths

The study utilizes a large amount of eye movement data, across a broad age range and multiple, albeit simple, tasks and shows that pretraining on this kind of general eye movement data improves the model's accuracy in clinical classification based on eye movements.

Critical weaknesses
  1. While the StimEM model is compared against CNN and Transformer baselines, the most important comparison against a baseline model based on eye movement features and demographics is missing. The authors themselves mention that feature-based classification is the standard in the field. Further, their ocular tokenizer is trained based on such features, so a comparison to the classification performance based on these features is obvious. Related to this, demographics are not included in the model or as a baseline comparison on the classification tasks.
  2. The origin of the data is not mentioned and all description pertaining to the origin of the datasets are highly vague. No mention of database / hospital etc. is highly problematic. If this was done for anonymity reasons, there should still be a way to provide information on the origin of the data. Further, there are no demographics reported for the clinical test cohorts, clinical and control groups may differ substantially in age or other features.
  3. It is not clear, whether different datasets were used during training and classification. The train/test splits only pertain to training and fine-tuning separately. I am thus concerned that data from the classification cohort was used during pre-training.
  4. Claims about representation learning and generalizability are overstated. In no way are learned representations evaluated. Further, the study possibly rests on a single dataset from one recording location and device, as well as the same set of 4 simple tasks. No generalizability can be claimed from this.
  5. Figures, text, and code all look heavily AI generated which together with the lack of data reporting gives me strong doubts about the trustworthiness of the presented results.
Questions for authors

What is the rationale behind the applied filtering with guard regions? Commonly the whole data is filtered. With the current filtering the model is implicitly informed about interesting regions.

Thank you for providing the code. To what extent was AI used to generate the code? Please add this do the AI Use Statement, following the ICLR guidelines.

How does the StimEM model size (number of parameters) compare to the size of the comparison models?

You describe the model as self-supervised, however the Ocular tokenizer is trained with trial level features, does that still classify the model self-supervised?

Reviewer_4TqVOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates learning representations from eye movements for clinical cognitive assessment. Prior works mostly use hand-crafted features or use small models that are trained with supervised learning, and unlabeled recordings would be wasted. This work instead propose StimEM that is pretrained on 500h recordings across 4 saccadic tasks. They first train an FSQ tokenizer using waveform reconstruction and the prediction of 36 behavioral features and designed a stimuls-conditioned mask such that eye tokens can attend to stimuli while stimulus tokends attend to themself, which looks like an interesting idea to me.

Critical strengths

The stimulus-conditioned attention with tokenizer-in-the-loop supervision are intuitive and results show evidences that are convincing with a 500h dataset and across six clinical tasks.

Critical weaknesses
  1. In Sec4.1 the authors claim that the pretraining and the downstream splits are built independently but the appendix A.3 calss that a transductive protocol. Looks like unlabelled data from downstream test subjects are included in pretraining.
  2. The tokenizer is trained with 36 behavioral features, but the paper did not try to do a direct comparison with a model that is directly on these features. This ablation layer is missing.
Questions for authors
  1. Could you please confirm whether there are test subjects leaked in the pretraining split? If so, how would the performance change without them?
  2. What is the accuracy you can achieve by training a model directly on the 36 features and comparing it to your work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.