acceptodds
Withdrawn from ICLR 2027

MindAU: EEG-Conditioned Facial Action Unit Editing via Dual-Stream Manifold Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent brain decoding studies have made substantial progress in reconstructing externally perceived visual content from neural signals. However, using electroencephalography (EEG) recordings to guide facial expression editing remains largely unexplored and poses a distinct challenge: rather than recovering what a subject sees, it requires identifying facial-action related patterns from noisy EEG signals and grounding them in localized, identity-preserving expression edits. In this paper, we investigate EEG-conditioned facial image editing for fine-grained facial action unit (AU) control and propose MindAU, a unified framework for controlling facial AU edits from EEG signals. MindAU first learns noise-robust and AU-discriminative EEG representations through temporal masked reconstruction and AU classification supervision. It then bridges the modality gap via Dual-Stream Manifold Alignment, aligning EEG features with AU-level text semantics and identity-reduced visual displacement trajectories in the multimodal space of Qwen2.5-VL. Finally, MindAU incorporates EEG-aware Multimodal Rotary Positional Embeddings, landmark-guided reference masking, and AU-aware region supervision into a multimodal diffusion-based editor for high-fidelity identity-preserving editing. We also introduce E-CAFE, a curated benchmark for EEG-Conditioned Action-Unit Facial Editing with paired EEG-Face editing samples and standardized evaluation protocols. Extensive experiments demonstrate the effectiveness of MindAU and suggest its potential as a step towards future assistive expression technologies for individuals with facial neuromuscular disorders.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain-computer Interfaces, Neural Decoding, Action-Unit Facial Editing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hwB1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MindAU for editing facial action units (AUs) from an EEG segment and a reference face. An AU-aware encoder learns signal representations, and dual-stream alignment connects them to AU text semantics and identity-reduced visual displacements. The resulting tokens condition a diffusion editor with reference masking and regional supervision. The authors curate E-CAFE from BU-EEG, containing 5,000 pairs across ten individually performed AUs from 28 healthy participants. Evaluation uses two held-out participants and also transfers their signals to synthetic reference identities. The method improves AU accuracy and identity similarity over the reported pipelines, with additional ablations and perceptual evaluation.

Critical strengths
  • A structured formulation with useful empirical support. AU-level supervision provides an interpretable link between biosignals and localized facial edits. The dual-stream design combines semantic alignment with visual displacement supervision, and the ablations support the value of AU supervision and supervised contrastive alignment. The curated benchmark and separate reference-identity protocol provide a useful starting point for further work.

  • Attention to shortcuts and limitations. Reference masking directly addresses the risk of copying the input expression: removing it raises identity similarity but reduces AU accuracy from 27.71% to 5.02%. Fixed-reference and EEG-free controls further examine signal dependence. The frequency/channel analysis and explicit limitations acknowledge unresolved EEG/EMG separation, which is valuable for interpreting the results.

Critical weaknesses
  • The neural interpretation and application motivation exceed the demonstrated setting. The experiments use recordings acquired while healthy participants actually perform facial actions. The original BU-EEG paper describes sensors covering the forehead and cheeks and explicitly retains EMG-like and EOG-like activity as useful information for facial analysis [1]. Thus, successful conditioning does not establish decoding of brain-origin facial intent. Appendix I.1 is helpful: after 0.1-50 Hz filtering, removing face-adjacent channels reduces Stage-1 classification accuracy from 32.93% to 28.95%. However, channel removal can also remove neural information, and residual muscle activity may remain. These classifier results do not establish the physiological source driving the final editor. Section 4.5 appropriately acknowledges this uncertainty and the absence of evidence for imagined or attempted movements. Nevertheless, the introduction still positions the contribution as control through brain activity and a step toward assistance for facial neuromuscular disorders. The demonstrated contribution should be framed consistently as editing from recordings during performed actions, with a clearer rationale for this acquisition setting. The title's use of EEG is defensible as a recording modality; the substantive concern is the gap between the neural-control motivation and the available evidence.

[1] X. Li et al. An EEG-Based Multi-Modal Emotion Database with Both Posed and Authentic Facial Actions for Emotion Analysis. arXiv:2203.15829, Sections II.B and III. https://arxiv.org/abs/2203.15829

  • The comparisons do not isolate the benefit of the proposed conditioning representation. Appendix B states that the LongCat-Image and FireRed editors remain frozen, whereas MindAU receives task-specific training with reference masking and regional losses. Using the same Stage-1 AU classifier controls the recognition component, but does not isolate the benefit of continuous EEG alignment from adaptation of the editor itself. The results establish an advantage over these off-the-shelf pipelines, while the stronger interpretation that continuous conditioning yields more faithful or intensity-aware control remains insufficiently tested. A baseline using the same editor, training pairs, and optimization with predicted AU labels or embeddings would materially strengthen the comparison. Calibrated intensity evaluation would also help substantiate the claimed advantage beyond AU-category recognition.
Questions for authors
  • Signal interpretation and scope. Which claims are intended to depend on brain-origin information? Please clarify the intended scope throughout the front matter. Is there any end-to-end generation analysis using the face-adjacent-channel exclusion or other artifact-reduction settings, beyond the Stage-1 classifier results?

  • Matched comparison. Can the authors provide a comparison using the same task-adapted editor and training losses, conditioned on predicted AU labels or AU embeddings? What evidence separates the benefit of continuous EEG conditioning, including expression intensity, from editor adaptation?

  • Signal specificity and robustness. Can the authors report paired versus within-participant, cross-AU shuffled EEG results while holding the reference and generation settings fixed? The zero-token control changes the input distribution substantially. Per-participant results for the two held-out subjects would also help assess whether gains are consistent across people.

  • Future validation. A future study of facial motor imagery or attempted movements, with facial EMG and video monitoring to check for overt or covert contraction, would directly test the assistive motivation. A new clinical study is beyond a reasonable rebuttal request; the immediate need is to align the claims with the demonstrated setting.

Reviewer_yNmrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper formulates the task of EEG-conditioned AU-target facial editing and proposes a three-stage framework to solve the task. The first stage pretrains a Masked Autoencoder to reconstruct the EEG signals from corrupted input, with an additional classification head for Action Units to encourage the features to associate with facial expressions. The second stage aligns the EEG embeddings to the text embedding space of the Qwen model using the description text of the AUs. The third stage injects the facial editing capability with additional losses on the general facial region and the facial region focused by the specific AU. A benchmark, E-CAFE, is also proposed with Self-Referenced and Cross-Referenced protocols. Results show superior performance, especially in terms of facial similarity.

Critical strengths
  • A new and well-motivated task formulated. In addition to predicting external stimuli with EEG signals, the authors use EEG signals to predict the subjects' internal facial expressions, which can be beneficial to the community, especially for disabled people with muscular difficulties.
  • Much higher performance than all the compared baselines, especially in facial similarity for face editing.
Critical weaknesses
  • Insufficient evaluation of the personalizability of the generated results. EEG is a highly personalized signal, and one crucial reason why one would use EEG signals rather than more straightforward and data-rich modalities such as the AU description or simply the AU class as conditioning is that EEG signals are highly personal and may be able to retain person-specific details and muscle movements that differ despite being conceived as the same Action Unit by other observers. Therefore, for an EEG-related paper for expression generation, it will be helpful to include the results of the same AU with different EEG driving signals to demonstrate the effect of the EEG modality, but it appears that such evaluation is not included in the paper.
  • Missing statistics for the proposed E-CAFE benchmark. Since the benchmark is considered one of the contributions of the paper, there should be a table introducing the number of samples it covers and its comparison with other datasets.
  • Hardcoded facial regions and their association with the action units may limit generalization. The authors defined facial-region masks for Lregion\mathcal{L}_{region} and Lfocus\mathcal{L}_{focus}, without showing a general automatable or principled pipeline for generating them. Therefore, the generalization of the work to other action units or non-AU facial expression descriptions may be non-trivial.
  • The reproducibility of the paper is concerning, as several details are not well documented, which I list below in the section "Questions for Authors".
Questions for authors
  • L191-L193 seems to be the only place where the term "teacher text tokens" is used. Are they the same AU-specific natural language descriptions as used to encode ztextz_{text} in Eq.4?
  • L212-L215 assumes the identity component of the latent vector to be collinear with the latent vector, while the expression component change is orthogonal. Is this proved theoretically or empirically in previous work or in your paper?
  • Are there examples or more details about Landmark-Guided Reference Masking? From the text L270-L280, it appears that the expression-rich part of the face is "masked"; are they replaced with uniform pixels or blurred? I am curious about this because they not only carry the information of the expression but also carry important details to maintain the identity of the reference image.
  • Are the focused masks (Eq. 17) and region masks (Eq. 15) the same masks? Do the masks adapt to different people based on their own face configurations, or do you use a general template?
  • The AU accuracy in Table 1 is much lower than the possible maximum that GT can achieve. Is that because the EEG was noisy or are there other analyses?
Reviewer_SH18OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes MindAU, which edits a reference face to show the facial action unit (AU) implied by an EEG segment. An EEG encoder is pre-trained with masked reconstruction and AU classification, then aligned to Qwen2.5-VL text embeddings and to image-displacement embeddings. The aligned encoder conditions a fine-tuned LongCat-Image diffusion editor that uses reference masking and region losses. The authors also build E-CAFE: 5,000 EEG-face pairs from BU-EEG (28 subjects, 10 posed AUs), with two held-out test subjects. MindAU outperforms DreamDiffusion and classify-then-edit pipelines in identity similarity, FID and AU accuracy (27.7%).

Critical strengths
  1. The paper probes its own confounds candidly. It includes a masking ablation that exposes the copy-the-reference shortcut, an EMG channel/frequency analysis, a fixed-reference control and an explicit limitations section. In the fixed-reference control, AU accuracy (27.5%) can only come from the recorded signal; this is the strongest evidence that the output depends on the input.
  2. A measurable task formulation. Identity-preserving editing in AU space is cleaner and more measurable than reconstructing perceived stimuli. The held-out-subject and cross-identity protocols are a reasonable starting point.
Critical weaknesses
  1. An EEG vs. EMG/EOG confound undermines the core claim. Participants perform posed AUs, and the 192 ms window overlaps the movement. The 0.1-50 Hz filter retains much of the EMG and essentially all of the EOG. Table 9 shows decoding depends strongly on face-adjacent channels and broadband content (47.9% -> 29.7%). There is no artifact removal, no window before movement onset, and no topographic attribution. Framing the work as brain decoding toward assistive technology is therefore not supported.
  2. The key causal control is missing. There is no matched-versus-shuffled EEG comparison or AU confusion matrix, which the authors themselves acknowledge. The zero-token "EEG-free" ablation is out of distribution (CSIM 25%, FID 136), so it measures breakage rather than the information carried by the EEG.
  3. The evaluation is too thin. It uses two of 28 subjects, does not report the test-set size, and gives no seeds, intervals or cross-validation. Most ablation differences are 0.5-2 points.
  4. The baselines are not matched. The pipeline baselines use frozen off-the-shelf editors, while MindAU is fine-tuned in-domain. The informative baselines are missing: the same fine-tuned editor with the predicted, or the ground-truth, AU label. MindAU's AU accuracy (27.7%) is even below that of its own classifier (32.9%), contradicting the motivation for continuous conditioning. The "intensity-aware" claims are never quantified.
  5. The metrics are weak. The 10-way conversion of a multi-label detector is unexplained, copying the reference scores 5.0% (below chance), and the absolute AU accuracy is about 37% of the ground-truth ceiling. The 30-case perceptual study ranks MindAU first on AU match 90-93% of the time, which is hard to reconcile with ~28% automatic accuracy.
  6. Limited novelty. The pipeline combines known parts: an MAE EEG encoder, a Q-Former-style adapter, supervised contrastive alignment and an MM-DiT editor. The "identity-reduced" trajectory removes only one direction along z_ref. E-CAFE is a filtered resampling of BU-EEG, and in training the reference frame is only about 5 frames before the target.
Questions for authors
  1. What is being decoded? Please report results after ICA/regression-based EMG/EOG removal, with low frequencies only (below 15-20 Hz), and with windows that end before movement onset. Please also show channel and frequency attribution maps.
  2. For the same references, what is AU accuracy with matched vs. AU-shuffled EEG? Please include the 10x10 confusion matrix of generated AU against EEG AU.
  3. Can you run leave-subjects-out cross-validation over all 28 subjects and report mean ± std, along with test-set sizes?
  4. How does the same fine-tuned editor perform when conditioned on the predicted AU label and on the ground-truth AU label?
  5. How is OpenGraphAU converted to 10-way accuracy, what are per-AU accuracy and chance level, and is the evaluator validated on the synthetic identities?
  6. How were the 30 perceptual-study cases selected?
  7. How often does a training reference already show the target AU? Were the Nano Banana 2 pairs generated only from training subjects?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.