acceptodds
Under review as a conference paper at ICLR 2027

PrivFusionNet: Mitigating Identity Re-Leakage in EEG–Eye Movement Fusion for Emotion Recognition

Abstract

Multimodal fusion leverages complementary signals but can also recombine residual subject-specific cues, causing identity information suppressed within individual modalities to re-emerge after fusion. Existing privacy-preserving approaches primarily operate at the unimodal level, leaving fusion-level privacy underexplored. Using emotion recognition from EEG and eye-movement signals as a testbed, we formulate fusion-induced identity re-leakage and show that unimodal identity suppression alone does not reliably protect fused representations. We propose PrivFusionNet, a privacy-preserving multimodal representation learning framework that directly protects the fusion space through three complementary mechanisms. Privacy-aware low-rank fusion reweights cross-modal interaction components according to task utility and identity risk; fusion-level task-identity disentanglement attenuates identity-related directions while retaining emotion-discriminative information; and a fused privacy codebook quantizes continuous representations into subject-shared task prototypes to suppress fine-grained identity cues. Under leave-one-session-out privacy evaluation, PrivFusionNet reduces fused-representation user identification accuracy from 25.33% to 7.57% on SEED-IV and from 46.38% to 6.11% on SEED-V. Under leave-one-subject-out emotion-recognition evaluation, it achieves mean accuracies of 84.23% and 88.30%, respectively. These results identify fusion-induced identity re-leakage as a practical privacy risk and demonstrate a favorable privacy-utility trade-off.

Primary area:
applications to neuroscience & cognitive science
Keywords:
multimodal representation learning, privacy-preserving learning, identity leakage, EEG, eye movement, emotion recognition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2Se9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PrivFusionNet to address identity leakage in EEG and eye movement fusion and applies it to emotion recognition. The method combines privacy-aware fusion, task–identity disentanglement, and a discrete codebook to suppress identity information in fused representations. The experimental results show that the method reduces user identification accuracy while maintaining emotion recognition performance, with additional validation through cognitive load recognition experiments.

Critical strengths
  1. The paper tries to address a practically relevant privacy issue, namely whether privacy protection applied separately to each modality remains effective after fusion.
Critical weaknesses
  1. The contribution is incremental and limited, as the main proposed modules are based on a combination of existing methods.
  2. The ablation experiments are unclear and should not be confined to the appendix. In addition to evaluating the effects of removing each module, the effects of using each module alone should also be evaluated. The paper also lacks ablations within each module. The effects of key module parameters, such as the number of rank components R in PA-LRF and the codebook size M in FPCB, should also be evaluated. The paper also does not explain why these parameter values were chosen.
Questions for authors
  1. The privacy evaluation uses a fixed classifier architecture and training budget, without examining how these choices affect the results. How sensitive are the reported privacy gains to these choices, and what limitations should be considered when interpreting the results?
  2. On SEED-IV, removing either PA-LRF or FTID produces only a small change in fused UIA. What evidence supports their complementary roles and the explanation that FPCB masks their contributions?
  3. The results show reduced identity leakage, but provide limited evidence explaining why the proposed modules suppress re-leakage. Can the authors provide experimental evidence to clarify how these modules work and what this tells us about the causes of re-leakage?
  4. Appendix B.2 states that the “best-performing model” is selected, but does not specify the validation split or selection metric. How is the validation set constructed from the training data, and which metric is used for model selection?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 2Se9 (Part 1/2: W1–W2 and Q1)

Thank you for recognizing the practical importance of evaluating whether unimodal privacy protection remains effective after fusion. Below, we clarify our contributions, the extended ablation design, parameter selection, and the scope of the attacker-based evaluation.

Response to W1: Contribution and novelty

Our core contributions lie in identifying fusion-level identity re-leakage and designing a privacy-protection framework tailored to this problem. We empirically investigate how identity information can become accessible after fusing EEG and eye-movement representations, even when identity information has been suppressed within each modality. This finding motivates explicitly evaluating and protecting the fused representation.

Our framework addresses different stages of this process. PA-LRF incorporates identity risk into rank-component weighting, allowing fusion to account for both task information and identity-leakage risk. FTID suppresses identity-related directions in the fusion space while preserving task information. FPCB constrains fused representations through subject-shared discrete prototypes to suppress fine-grained identity cues. These mechanisms are jointly optimized under task and privacy objectives.

To assess the added value of these designs, we have added comparisons with a fusion-level GRL baseline and configurations containing individual modules or module combinations. Reporting both BCA and UIA allows us to evaluate their contributions in terms of identity suppression and task-information preservation.

Response to W2: Ablation design and parameter choices

The original Table 5 evaluates each module's marginal contribution when the other two remain active. We have extended this comparison with configurations that introduce only PA-LRF, FTID, or FPCB to the same backbone with fusion-level GRL, alongside the existing two-module combinations and the complete model. All configurations follow consistent training and evaluation protocols, with both BCA and fused UIA reported. These comparisons distinguish individual module effects from the benefits of combining modules. The accompanying analysis is provided in Appendix C.

Regarding parameter selection, R controls the representational capacity of low-rank fusion in PA-LRF, while M controls the codebook capacity and degree of representation compression in FPCB. Both influence the balance between task-information preservation and identity suppression. Their values are selected using validation performance within the training partition.

For example, in a LOSO fold on SEED-IV, one subject is held out for testing, and the remaining 14 subjects are divided into 13 training subjects and one validation subject. After selecting the parameter configuration, the validation subject's data are merged into the training partition for final training on all 14 source subjects. The outer test subject is excluded from parameter selection and model training.

Response to Q1: Attacker architecture and training budget

Our privacy evaluation follows the identity-classifier-based evaluation paradigm used in studies such as ID-RemovalNet. We independently train a nonlinear MLP identity probe on fixed representations and use the same attacker architecture, training budget, and data splits across all methods in the privacy comparison. This provides a controlled basis for comparing identity leakage across methods.

Attackers with greater capacity, different architectures, or more extensive training may recover additional identity information.

Accordingly, our conclusions concern identity-leakage mitigation in the final fused representation under the evaluated identity attacks. Near-chance UIA under this protocol should not be interpreted as a guarantee that all identity information has been removed.

AuthorsOpenReview ↗

Response to Reviewer 2Se9 (Part 2/2: Q2–Q4)

Response to Q2: Small ablation effects and complementarity

The small UIA changes on SEED-IV measure the marginal effects of PA-LRF and FTID when the other modules remain active. In the complete model, FPCB already reduces final fused UIA to near-chance levels, leaving limited room for further measurable reductions. This provides a possible explanation for why removing PA-LRF or FTID produces only a small change in final UIA; the removal results alone do not establish their standalone contributions or demonstrate that FPCB masks their effects.

To examine their contributions more directly, we have added configurations that introduce only PA-LRF, FTID, or FPCB to the same backbone with fusion-level GRL, reporting both BCA and UIA. Together with the existing two-module configurations, these comparisons evaluate the effects of adding PA-LRF or FTID with and without FPCB. They therefore allow us to assess both identity suppression and task-information preservation, rather than inferring complementarity solely from small changes in final UIA. The corresponding results and analysis are provided in Appendix C.

Response to Q3: Evidence for how the modules work

Our working explanation is that complementary residual identity cues can remain in individually protected modalities. Jointly accessing these cues through fusion may make identity information more accessible than it is in either modality alone.

The proposed modules target different stages of this process. PA-LRF downweights rank components with high identity risk during fusion, reducing the contribution of identity-related cues. FTID suppresses identity-related directions after fusion, limiting their retention in the continuous representation. FPCB then constrains fine-grained differences between individuals through subject-shared discrete prototypes. Together, these mechanisms aim to reduce identity leakage while preserving task-relevant information.

The extended module-level ablations in Appendix C enable or disable these mechanisms under matched conditions and report both BCA and UIA. These experiments assess each mechanism's empirical contribution and the effects of their combinations. They provide evidence about how the proposed design affects leakage and utility, while the explanation based on complementary residual identity cues remains a mechanistic interpretation rather than a uniquely established cause.

Response to Q4: Validation split and model selection

The validation set is constructed entirely within the outer training partition. The outer test data are excluded from both model selection and training.

For example, in a LOSO fold on SEED-IV, we first hold out one subject for testing and divide the remaining 14 source subjects into 13 training subjects and one validation subject. During model selection, model weights are updated using only the training subjects' data, and validation performance is used to select the configuration.

After model selection, we merge the validation subject's data into the training partition and complete final training for the predefined 80 epochs using all 14 source subjects under the selected configuration. Validation data therefore participate in final training only after model selection is complete. The outer test subject remains independent and is used solely for final evaluation.

The revised Appendix B.2 describes the validation-set construction and final training procedure, distinguishing validation-based selection from final training on the combined training and validation data.

Thank you for the responses and revisions. My main concern remains that the privacy evaluation relies on a single type of identity classifier. The new Table 7 shows that representations with near-chance UIA can become highly identifiable once simply combined, so near-chance UIA from one per-sample probe is weak evidence of identity removal. Identity may still be revealed by aggregating the patterns of codeword use across multiple samples, which the current evaluation does not test. In addition, the framework largely builds on existing EEG methods (e.g., CMSLNet, ID-RemovalNet, MIND-EEG), so its methodological novelty remains limited. I therefore maintain my rating.

AuthorsOpenReview ↗

Response to Reviewer 2Se9 (Follow-up: Methodological Contributions and Privacy Evaluation)

Thank you for raising this concern. We would like to clarify that our methodological contribution does not lie in introducing entirely new primitives, but in formulating and addressing a distinct privacy failure mode at the multimodal fusion level. Specifically, PA-LRF introduces identity-risk-aware weighting into low-rank cross-modal interactions through α=softmax(u−βρ)\alpha=\mathrm{softmax}(u-\beta\rho); FTID explicitly learns an identity-related subspace in the fused representation space and suppresses it through projection, which is fundamentally different from IDS, where identity suppression relies on adversarial training with gradient reversal at the unimodal level; and FPCB combines subject-shared vector quantization with subject-conditional code-usage regularization. These mechanisms differ in their mathematical formulations, optimization objectives, and positions in the processing pipeline, operating at complementary stages of cross-modal interaction, continuous representation, and discrete representation. Their joint design therefore constitutes a fusion-oriented privacy framework rather than a direct reuse of existing EEG privacy methods.

Regarding cross-sample codeword aggregation, we acknowledge that it represents a stronger attack setting than the per-sample identity probe used in our evaluation. However, privacy protection and identity inference inherently involve an evolving interplay between defense and attack capabilities. Our objective is to identify fusion-induced identity re-leakage and establish a systematic defense against identity attacks under a controlled and commonly adopted evaluation protocol, rather than claim immunity to every conceivable attacker. Importantly, FPCB already incorporates subject-conditional code-usage regularization to reduce subject-specific code-selection preferences. More advanced aggregation-based attacks can motivate corresponding improvements to privacy defenses. We view this as an iterative research process: as attack capabilities advance, privacy mechanisms and their evaluations should evolve accordingly, rather than treating the existence of increasingly powerful attacks as invalidating the value of the current defense.

Reviewer_8k1qOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies "fusion-induced identity re-leakage," where multimodal fusion can recover subject identity even after identity information has been suppressed in individual EEG and eye-movement representations. PrivFusionNet tackles this with privacy-aware low-rank fusion, fusion-level task-identity disentanglement, and a fused privacy codebook. Experiments on SEED-IV and SEED-V examine fused identity leakage and cross-subject emotion recognition, with an extra extension to cognitive-load recognition on CL-Drive. The method cuts fused user-identification accuracy substantially while keeping task performance competitive.

Critical strengths
  1. The privacy failure mode is interesting and easy to overlook, and it is practically relevant that individually protected modalities can still become identity-discriminative once fused. The phenomenon shows up on SEED-IV, SEED-V, and a cognitive-load dataset, which makes the motivation solid.

  2. The evaluation covers both sides well, with representation-level identity attacks alongside emotion recognition under cross-session and LOSO protocols, plus ablations for the privacy modules, which is reasonably comprehensive for this EEG-eye setting.

Critical weaknesses
  1. The main verification uses ACM + Att-LMF after unimodal identity suppression, but simple concatenation or substantially different fusion operators are not tested, so is the leakage really induced by learned fusion, or does it just come from jointly exposing complementary residual identity cues from two modalities?

  2. PA-LRF, subspace-based disentanglement, adversarial suppression, and vector quantization look largely like adaptations of existing techniques, and the ablation shows removing PA-LRF or FTID barely changes privacy while FPCB accounts for most of the UIA drop, so shouldn't simpler fusion-level privacy baselines be compared?

  3. Privacy is mainly evaluated with one MLP identity probe, yet the intermediate EEG and eye representations stay strongly subject-identifiable even when the final fused UIA is near chance, so doesn't the privacy claim hinge on both attacker capacity and the assumption that intermediate representations can't be accessed?

  4. Privacy uses leave-one-session-out splits where the same subjects appear in both training and testing, while utility uses LOSO on unseen subjects, so do the experiments really show the privacy protection generalizes to unseen identities, given that cross-subject generalization is a central utility claim?

  5. The strongest task baselines, including CMSLNet, are taken from prior work under a "compatible" rather than identical pipeline, and PrivFusionNet improves over CMSLNet by only about 1 point, with a fixed random seed and no uncertainty estimates, so how are we supposed to tell whether these small gains and ablation differences are statistically meaningful?

Questions for authors

Please see Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 8k1q (Part 1/2: W1–W2)

Thank you for recognizing the relevance of the privacy failure mode and the breadth of the evaluation. We clarify the effects of different fusion strategies, the contributions of the proposed modules, and the scope of our privacy and utility evaluations below.

Response to W1: Joint exposure versus learned fusion

Our original verification examines whether unimodal privacy protection remains effective after fusion. We use a single competitive backbone, ACM + Att-LMF, consistently across the compared settings to control for differences in fusion architecture when evaluating privacy protection and task performance.

To ensure that our observations are not specific to the choice of learned fusion operator, we have expanded the comparisons in Appendix D to include direct concatenation and representative alternative fusion operators, using the same datasets and consistent training and evaluation protocols. We primarily examine fused UIA and also report BCA to assess identity leakage alongside task performance.

Direct concatenation provides a reference setting in which the protected unimodal representations are jointly exposed without an additional learned fusion operator. Comparing this setting with learned fusion allows us to examine how different fusion strategies affect the accessibility of residual identity cues. These additional experiments broaden the evaluation of identity re-leakage beyond the original ACM + Att-LMF backbone and further characterize the conditions under which unimodal privacy protection remains effective after fusion.

Response to W2: Existing techniques and simpler alternatives

Our contribution lies in identifying the potential failure of unimodal privacy protection after fusion and developing a framework that balances identity suppression with task-information preservation in the fused representation. The modules address different stages of this process: PA-LRF incorporates identity risk into rank-component weighting during fusion, FTID suppresses identity-related directions in the fused representation while preserving task information, and FPCB constrains the representation through subject-shared discrete prototypes.

To clarify the added value over simpler alternatives, we have added a fusion-level GRL baseline to Table 1, retaining the same ACM + Att-LMF backbone while disabling PA-LRF, FTID, and FPCB. We report both BCA and UIA under matched training and evaluation protocols.

The small UIA changes observed when removing PA-LRF or FTID measure their marginal effects when the other modules remain active, rather than their standalone effectiveness. Their contributions should therefore be assessed jointly through identity leakage and task performance. Removing FPCB produces the largest increase in fused UIA, indicating its central role in identity suppression within the evaluated architecture.

To further examine the individual contributions and the benefits of module combinations, we have extended Table 5 with configurations that add only PA-LRF, FTID, or FPCB to the same fusion-level GRL baseline, alongside the existing two-module combinations and the complete model. We have also expanded Appendix C (Ablation Analysis) to analyze these results and characterize the privacy–utility performance of the individual modules and their combinations.

AuthorsOpenReview ↗

Response to Reviewer 8k1q (Part 2/2: W3–W5)

Response to W3: Attacker capacity and intermediate representations

Our work builds on studies such as ID-RemovalNet that evaluate identity leakage using identity classifiers, and extends this evaluation to identity re-leakage after multimodal fusion. To ensure comparability across methods, we use the same MLP identity probe and evaluation protocol for all compared methods, controlling for differences in attacker configurations.

Our intended deployment scenario keeps raw data and intermediate EEG and eye-movement representations within a trusted processing environment, while exposing only the protected final fused representation for downstream use. Accordingly, our privacy objective concerns this final released representation. The high intermediate-representation UIAs in Table 1 illustrate this protection boundary; access to these intermediate features falls outside the scope of protection under this setting.

A common attack protocol supports controlled comparisons across methods, while the resulting privacy conclusions remain specific to the evaluated attacks. Stronger attackers may recover additional identity information. We therefore interpret the results as evidence of identity-leakage mitigation under the evaluated protocol, rather than equating near-chance performance of a single probe with the complete removal of identity information.

Response to W4: Generalization to unseen identities

Our privacy evaluation and cross-subject task evaluation address two distinct objectives: protecting data providers’ identities in released representations and preserving task utility for unseen subjects.

Specifically, our privacy objective concerns users whose data contribute to dataset construction and protection-model training. We suppress identity information in their fused representations before making these representations available for downstream tasks. The leave-one-session-out privacy protocol evaluates whether this protection remains effective for these users in held-out sessions.

The LOSO task protocol evaluates whether the learned representations preserve sufficient task-relevant information for emotion recognition to generalize to unseen subjects. Accordingly, cross-subject generalization refers to downstream task utility, while the privacy results establish identity-leakage mitigation for participating users across sessions.

Response to W5: Baseline comparability and uncertainty

Our primary objective is to mitigate identity leakage in fused representations while preserving competitive downstream task performance. The comparison with CMSLNet and other strong baselines in Table 2 provides context for assessing task utility under privacy protection. We do not interpret the numerical improvements of 1.08 and 0.98 percentage points as evidence of statistically significant superiority.

As stated in Appendix B.4, the CMSLNet results are taken from prior work under a compatible within-session LOSO protocol, rather than reproduced within our implementation. This comparison therefore provides a reference for task performance, rather than an identical-pipeline comparison.

Appendix B.5 specifies that a fixed random seed was used to facilitate reproducibility. Our reported results are averaged across all evaluation folds, rather than obtained from a single train–test split. Accordingly, we frame our contribution around identity-leakage mitigation with competitive task utility, without claiming statistically significant task-performance gains.

Thank you for the detailed rebuttal and substantial revisions. I appreciate the additional experiments and clarifications, which have addressed most of my original concerns. I will therefore raise my score.

Reviewer_W2c3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates identity re-leakage in EEG–eye movement fusion for emotion recognition, showing that unimodal privacy protection may be insufficient. PrivFusionNet combines privacy-aware low-rank fusion, task–identity disentanglement, and a fused privacy codebook to suppress identity information while preserving task utility. Experiments on SEED-IV and SEED-V show substantially reduced user identification accuracy while maintaining competitive emotion recognition performance.

Critical strengths

The paper provides useful empirical evidence that suppressing identity information in individual EEG and eye-movement representations does not necessarily prevent identity inference after fusion. This finding highlights a practical limitation of unimodal privacy protection and motivates explicit privacy evaluation and protection at the fusion level.

Critical weaknesses

(1) The privacy evaluation lacks simple fusion-level baselines, particularly standard fusion followed by GRL-based identity adversarial training. It remains unclear whether the proposed three-module architecture provides substantial benefits over simpler privacy-preserving approaches.

(2) Table 5 provides limited evidence for the effectiveness of PA-LRF and FTID, as removing them increases UIA on SEED-IV by only 0.05 and 0.09 percentage points, respectively.

(3) The experimental section in the main text is limited and would benefit from additional results to more thoroughly validate the proposed method.

Questions for authors

Please refer to Q1 and Q2 in the “Critical Weaknesses” section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer WCC3

Thank you for recognizing the empirical evidence for identity re-leakage and the motivation for explicit fusion-level protection. We agree that the most useful additions are a simple fusion-level adversarial baseline and clearer evidence for the individual components. We address these points below.

Response to W1: Simple fusion-level privacy baselines

Table 1 in the original submission primarily examines whether unimodal identity suppression remains effective after fusion, and therefore does not directly include a fusion-level GRL baseline. We agree that this comparison is important for assessing the additional value of the three modules beyond direct identity-adversarial training. We have extended Table 1 in the revised manuscript with an “ACM + Att-LMF + fused GRL” baseline for both SEED-IV and SEED-V. Under matched data splits, model-selection procedures, and attack-evaluation protocols, we report both task BCA and identity UIA to directly compare its privacy protection and task performance with those of the complete method.

Response to W2: Small PA-LRF and FTID effects

Table 5 evaluates the effect of removing one module while retaining the other two. Thus, the 0.05 and 0.09 percentage-point UIA increases on SEED-IV reflect marginal effects within the complete architecture, rather than standalone effectiveness.

These ablations also provide evidence of robustness to individual module removal: the remaining configurations retain broadly comparable task performance and lower identity leakage than the unprotected baseline. Removing FPCB produces the largest increase in fused UIA, indicating its central role in identity suppression. PA-LRF and FTID are designed to balance identity suppression with task-information preservation.

To validate the additional benefits of combining the modules for privacy protection and task performance, we have extended Table 5 with configurations that add only PA-LRF, FTID, or FPCB to the same fusion-GRL baseline. The revised table reports BCA and UIA under matched conditions. We have also expanded Appendix C (Ablation Analysis) to summarize the additional results and analyze the individual module contributions and the privacy–utility performance of their combinations.

Response to W3: Experimental evidence in the main text

Thank you for this suggestion. We have strengthened the main-text experiments by adding the fusion-level GRL baseline to Table 1. We have also expanded the ablation results in Table 5 and the corresponding analysis in Appendix C to provide a more complete evaluation of the individual modules and their combinations.

Comments on the paper

1 comment

Dear Reviewers,

As we are approaching the end of the discussion period, I would like to kindly remind everyone to review the authors' responses and follow up on any outstanding questions or concerns. If you have not yet engaged in the discussion, please take a moment to do so before the deadline.

Thank you all for your time and efforts throughout the review process! Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.