acceptodds
Withdrawn from ICLR 2027

Brain2Face: A Study on the Relation of EEG Signals and Spatio-temporal Head Avatars

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Driving digital head avatars is classically done by hand-crafted artist design or by image-based facial performance capture. In this work, we ask whether spatio-temporal tracking of the humans's head geometry and appearance can be performed solely from brain signals, i.e., Electroencephalography (EEG). Since prior work mostly focused on activity and emotion recognition, this task is relatively unexplored without even having publicly available high-quality datasets and benchmarks. Thus, we contribute a dataset featuring paired EEG and human head geometry as well as appearance annotations. We synchronize 64-channel EEG with per-frame expression latent codes of a personalized Gaussian head avatar. Second, we introduce a baseline to the task of expression recovery of a head avatar from EEG signals. To this end, we propose a dedicated EEG encoder that takes a temporal sequence of EEG signals as input and regresses the latent expression parameters of the 3D Gaussian head avatar. Lastly, we validate our capture and algorithmic design choices through a benchmark protocol that we introduce to this task. We believe this work paves the way towards better understanding of the relation between EEG signals and facial surface as well as appearance changes. Code and data will be made publicly available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
EEG, avatars, datasets

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yYnsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether the facial expressions of a personalized 3D Gaussian head avatar can be decoded from scalp EEG. Six participants perform 75 posed expressions while wearing a 64-channel EEG cap inside a multi-view camera rig, and the avatar’s per-frame expression codes serve as targets. A REVE encoder with a small convolutional decoder maps one-second EEG windows to these codes. The experiments compare EEG foundation models, initializations and channel masks, with models trained and tested within each participant.

Critical strengths

The paired setup is rare: 64-channel EEG synchronized with 26–33 4K cameras, with targets in an avatar latent space, so results can be evaluated in code, mesh and image space. At a larger scale and with good documentation, this would be a useful resource.

Critical weaknesses

W1. Six participants are too few for a dataset contribution

All six participants are male, and the expression data total about 74 minutes, roughly 12 minutes per person (Tab. 1, A.2). No participant is held out for testing (Sec. 5.1). The encoder and depth comparisons use a single participant (Tabs. 2–3), and the six-person averages in Tabs. 4–5 are given without any spread. With this sample I cannot tell whether the conclusions hold beyond these six people, and since the dataset is the first contribution, this is the main reason for my rating.

Recording more people is feasible: EEG is non-invasive, the protocol took about 42 minutes of capture per person (Tab. 1), and recent EEG–video datasets are much larger, e.g., EAV (42 participants; Lee et al., Scientific Data 2024), a mixed-emotion dataset with EEG and face video (73 participants; Yang et al., Scientific Data 2024), EEG2Video (20 participantsLiu et al., NeurIPS 2024) and EgoBrain (40 participants; Lin et al., ICLR 2026). It would help to discuss these paired EEG–video datasets in the related work and make clear what the new dataset adds. I would encourage the authors to record at least 30 participants, add a held-out-participant split, and resubmit.

W2. No control for muscle artifacts, and no trivial baseline

The paper asks whether expressions can be recovered "solely from brain signals" (abstract). However, the kept band (0.5–99.5 Hz) includes the range where muscle artifacts are strong, and apart from amplitude clipping, the preprocessing removes no artifacts (Sec. 4.1). The paper's own observations point the same way: jaw clenches are used for synchronization because they cause large EEG artifacts (A.1), and for seen expressions, masking the temporal or frontal channels hurts most (Tab. 5). The authors themselves note that these channels lie closest to the temporalis and frontalis muscles, where EMG is strongest (Sec. 5.5). There is also no trivial baseline such as predicting the mean face, so I cannot judge how large the effects are.

Questions for authors

Questions marked clarification ask for information that should already be available; those marked additional analysis can be done with the existing recordings.

Q1. Cross-participant generalization (additional analysis)

With the current data, how well does a model trained on five participants work on the sixth (leave-one-subject-out)? The expression codes are disentangled from identity (Sec. 3), so this should be possible without new recordings. Please also report per-participant results, or the spread across the six participants, for Tabs. 4–5.

Q2. Trivial baselines (additional analysis)

What do simple baselines score on the metrics of Tabs. 4–5, for example predicting the mean expression code of the training set, or the neutral face? Without them, it is hard to tell how much of the reported performance comes from the EEG.

Q3. Muscle and eye artifacts (additional analysis)

How much of the performance remains (a) after low-pass filtering below 15–20 Hz, or (b) after removing muscle and eye components with ICA (e.g., ICLabel)? And (c) how does a simple model that uses only high-frequency (>30 Hz) band power, which mainly reflects muscle activity, compare with the full model? These controls would show whether the model decodes brain activity or facial EMG.

Q4. Subtle versus large expressions (additional analysis)

Many of the 75 expression types are large movements, and ten of them involve the tongue (App. B). Do the errors, relative to the mean-face baseline, differ between subtle expressions (e.g., raising an eyebrow) and large ones (e.g., jaw or tongue movements)? Per-expression-type errors from the existing models would be enough.

Q5. Model selection and the proposed decoder (clarification)

REVE and L = 2 were chosen on Subject-03 test sequences (Secs. 5.2–5.3), and Tab. 6 lists no validation split. Was any validation data used to choose the encoder, the decoder depth or the checkpoints, and how many seeds were run? Also, with the encoder frozen, the MLP head in Tab. 2 has lower LPIPS (0.0575 vs. 0.0643), blendshape MAE (0.0238 vs. 0.0434) and mesh MAE (0.725 vs. 1.074 mm) than the proposed decoder with L = 2 in Tab. 3, although it is worse on latent MSE and the landmark loss. Do the two tables use the same input window ("the full EEG input window" in Sec. 5.2 vs. one-second windows in Sec. 5.3) and the same training budget?

Q6. Timing of the demonstration videos (clarification)

Participants perform each expression after a video demonstration (Sec. 3.1). Was the video still on screen during the five-second segments, and how were the expression onsets annotated? For seen expressions, masking the occipital channels also hurts (Tab. 5), so responses to the video itself could contribute to the decoding.

Q7. Data release and ethics (clarification)

What exactly will be released: the full 4 h 15 min of recordings or only the 891 segments, and which of raw EEG, multi-view video and fitted avatars? Under what license and access controls? Did the participants consent to the public release of identifiable face video together with EEG, and was the study approved by an ethics board?

Reviewer_h1LAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Brain2Face, which uses 64-channel EEG to drive personalized 3D Gaussian head avatars. The authors collected multi-view videos and synchronized EEG data from 6 participants across 75 different facial expressions, first fitting the UniGAHA general-purpose Gaussian head model to each subject’s data to obtain an identity code Zid∈R512Z_{id}\in\mathbb{R}^{512} and a frame-by-frame expression code Zexp∈R256Z_{\mathrm{exp}}\in\mathbb{R}^{256}; Next, using EEG-based models such as REVE combined with a convolutional residual mapper, the EEG windows were regressed onto the facial expression code sequence, supervised by three types of losses: latent code, mesh, and landmark. The paper reports experiments on encoder comparisons, decoder depth, pretraining effects, and electrode ablation in brain regions.

Critical strengths
  1. This is the first study to perform continuous regression of EEG signals onto the facial expression latent space of a Gaussian head avatar. The task formulation is novel, and the study includes a synchronized EEG–multi-view facial dataset with dense facial expression annotations.

  2. The study systematically compares nine EEG baseline models with three initialization methods (freezing, fine-tuning, and training from scratch). The ablation analysis is comprehensive, providing a useful starting point for future work.

Critical weaknesses
  1. Core claim not validated: While the title states that the study investigates the “relationship” between EEG and facial movements, the main text only reports regression reconstruction accuracy and lacks analysis of frequency bands, time delays, or correlations; thus, the term “relationship” is misleading.

  2. Lack of key controls to rule out EMG artifacts: The frontotemporal electrodes are located in close proximity to the frontalis and temporalis muscles, so the model may primarily be decoding muscle artifacts; the paper does not include a lower bound for shuffled EEG, an EMG control, or a negative control, while the significant performance drop observed when the temporal lobe was masked during brain region ablation is consistent with the artifact hypothesis.

  3. Insufficient experimental scale to support “benchmark” status: The in-depth comparison of encoders and decoders was conducted solely on Subject-03; while the study spanned two sessions, both sessions were used for training, and transfer to new subjects or entirely new sessions was not evaluated.

  4. Limited methodological innovation: The approach essentially involves fitting an off-the-shelf EEG-based model to an existing avatar latent space via regression; the primary contribution lies in the dataset, which consists of only 6 participants and is therefore relatively small.

  5. Lack of Ethical Details: The authors commit to publicly releasing identifiable facial avatars, identification codes, and EEG data, but fail to specify whether separate informed consent was obtained for public release, the method used for EEG de-identification, or access controls; a dual-use risk assessment is also missing.

Questions for authors
  1. Can you supplement the analysis with lower-bound experiments using shuffled EEG or temporally misaligned EEG to demonstrate that the model has learned genuine temporal correspondences rather than overfitting?
  2. Are there any control experiments with synchronous EMG recordings or experiments capable of regressing out muscle artifacts? How do you rule out the competing explanation that the model is “decoding facial muscle artifacts”?
  3. Aside from Subject-03, what are the results of the encoder comparisons across the remaining subjects? Can you provide additional zero-shot/few-shot results across subjects or from new sessions?
  4. What are the statistics for the number of trials, total duration, and EEG signal-to-noise ratio for each expression per subject in the dataset? Can the specific sequence list used for segmentation be made public?
  5. When releasing the data, did the subjects provide separate consent for the public release of recognizable facial avatars and identification codes? How was the EEG de-identified, and is the data available for open download or via controlled access?
Reviewer_AjPqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper works on the task of predicting expressions from EEG signals. To solve the task, the authors first collected a multi-view dataset of 6 subjects with 75 expressions to gather synchronized EEG-face data. Then, the authors proposed a model to generate Gaussian Splatting-based head avatars based on EEG signals. Specifically, a pretrained backbone encodes the EEG data into features, which are then decoded into the expression codes of the UHAP model by a model trained with MSE and landmark Wing losses. Comprehensive ablations of the hyperparameters are presented.

Critical strengths
  • The dataset is comprehensive and beneficial to the community if released publicly. The current research in EEG-to-avatar generation is bottlenecked by the extreme scarcity of available datasets with synchronized facial captures and EEG data, especially multi-view face captures. The 64-channel EEG data improves greatly upon the previous 16-channel data.
  • The comparative study of EEG foundation models on a high-dimensional output is itself helpful. Although there were no baselines for reference, Table 2 provides a comparison among a variety of foundation models, which can further become a benchmark on over the foundation models to understand information of high granularity.
Critical weaknesses
  • The in-text citations are clearly in violation of Section 4.1 citations within the text of the Formatting Instructions for ICLR 2027 Conference Submissions, which mandates, "Otherwise, the citation should be in parentheses using \citep{} (as in “Deep learning shows promise to make progress towards AI (Bengio & LeCun, 2007).”)."
  • The authors frame their work as the first study of its kind (L77), but in the discussion of L156-L161, it seems that this work only extends the previous work in the dataset scale, with a minor difference in the expression representations, but they are both generating 3D head avatars based on EEG signals.
  • There is no baseline for comparison in the benchmark or the result presentation. There are comparisons in the tables of the paper, but all the entries are obtained from the proposed method without a previous work to compare with to understand the effectiveness of the proposed method. Even if there are few predecessors for comparison, a trivial baseline and an oracle-informed baseline should be included so that the readers can have a sense of the possible ranges of the metrics by chance or on GT images of a person at different expressions.
  • Limited number of subjects in evaluation. Table 4 used all 6 subjects, which can be acceptable given the fact that EEG data are expensive and the benchmark contains only 6 subjects, but there is no clear reason why Tables 2 and 3 used only a single subject.
Questions for authors
  • During the rebuttal period, please make sure you are familiar with the LaTeX \citep and \citetcommands to at least avoid duplicate author names in L112 and L115, and proofread your manuscript before you upload a revised version.
  • Can you replace Table 2 and 3 with the results evaluated on all subjects?
  • Will the Brain2Face dataset be made publicly available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.