acceptodds
Under review as a conference paper at ICLR 2027

MNCFR: Embodied Simulation-Inspired Reaction Formation and Memory-Augmented Regulation for Multiple Appropriate Facial Reaction Generation

Abstract

In dyadic human–human interactions, observing a speaker's emotional expressions may elicit emotional contagion and facial mimicry in the listener, with embodied simulation considered a potential underlying mechanism. Existing Multiple Appropriate Facial Reaction Generation (MAFRG) methods typically learn direct mappings from the given speaker behavior to multiple appropriate facial reactions (AFRs), without explicitly modeling the perception-to-response transformation. Inspired by embodied simulation, we propose Multimodal Neural Context Formation and Regulation (MNCFR), a novel diffusion framework comprising two complementary components: the Reaction Formation Module (RFM) and the Memory-Augmented Regulation Module (MAR). Specifically, RFM uses listener-specific information derived from listener reaction history to transform the speaker's currently perceived multimodal behavior into an intermediate listener-side reaction representation and then into a generation condition for the current window. This explicit speaker-to-listener transformation models the perception-to-response pathway proposed in embodied simulation. The same intermediate representation is also fed to an auxiliary EEG prediction head, which predicts segment-aligned listener EEG and provides an additional listener-side training signal. Meanwhile, MAR applies diffusion-step-dependent gating to a bounded-memory readout of recent speaker representations and injects the resulting context into the RFM condition, thereby promoting smoother transitions between the generated listener facial reactions across consecutive time windows. By separating current reaction formation from recent-context regulation, MNCFR organizes complementary information across temporal scales to support coherent reaction generation. Experiments under the official offline and online protocols on MARS show that MNCFR achieves better overall performance than reported learning-based methods. Ablation and temporal analyses demonstrate the effectiveness of RFM and MAR, while EEG prediction analysis further shows that higher prediction quality based on RFM's intermediate listener-side reaction representation is associated with larger generation gains.

Primary area:
generative models
Keywords:
Multiple Appropriate Facial Reaction Generation, Listener Facial Reactions, Embodied Simulation, Multimodal Interaction, Diffusion Models, Temporal Context Modeling, EEG-Guided Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7owKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

MNCFR proposes a diffusion framework for facial reaction generation inspired by embodied simulation. RFM maps the speaker's multimodal behavior and the listener’s past reactions into listener-side conditioning representations, with auxiliary EEG supervision, while MAR uses short-term memory and diffusion-timestep gating to improve temporal continuity. The method achieves high reaction appropriateness on MARS (offline FRCorr: 0.802; online FRCorr: 0.715), but some baseline values are inconsistent with the cited sources, and its diversity and synchrony scores are not the best. The main concerns are that the core speaker-to-listener transformation lacks an explicit mathematical definition, and that the correlation between EEG prediction performance and generation quality does not establish that the model implements embodied simulation. The mechanistic interpretation therefore appears somewhat overstated.

Critical strengths
  1. The framework separates current reaction formation (RFM) from historical context modulation (MAR) and incorporates auxiliary EEG supervision, providing a clear modeling approach for multimodal facial reaction generation.

  2. The paper includes offline and online evaluations, component ablations, temporal smoothness analysis, and subjective perceptual evaluation. Multiple metrics demonstrate the method's potential advantages in reaction appropriateness and temporal continuity.

Critical weaknesses
  1. The motivation for explicitly modeling the transition from perception to reaction is not sufficiently clear. The paper does not establish which aspects of this transient process existing methods fail to capture, how substantial these limitations are, or how they affect the quality of generated reactions.

  2. The caption of Table 1 states, "Published baseline values follow the REACT 2025 report." However, several reported baseline values appear inconsistent with the sources. For example, the original REGN paper reports an FRDist of 141.93, whereas Table 1 reports 155.49. The benchmark version is also insufficiently specified. REACT 2025 introduced the MARS dataset, while REACT 2026 extended the tasks to include EEG prediction and personalized generation. The paper does not clearly identify the exact official results used or reconcile the numerical discrepancies across sources. These ambiguities make the reliability and comparability of the reported results difficult to assess.

  3. The function gg in Eq. (7), which represents the central speaker-to-listener transformation, is not explicitly defined. In particular, the mathematical correspondence between gg and components such as the Mirror Mapper and FiLM Generator is unclear. This leaves the computation underlying the method’s central transformation insufficiently specified.

  4. Eq. (11) defines the gating coefficient as ατ=f(eτ)\alpha_\tau=f(e_\tau). If eτe_\tau encodes only the diffusion timestep and ff receives no additional sample-specific inputs, samples at the same timestep share the same gating coefficient. The coefficient is therefore not explicitly conditioned on the current speaker behavior, listener identity, or memory content. Although the modulated representation rtr_t can still differ across samples, the gate itself appears to implement diffusion-timestep-conditioned context scaling.

Questions for authors
  1. What evidence supports the need to explicitly model the perception-to-reaction transition? Which specific aspects of this brief transition are inadequately captured by existing methods, and to what extent does this limitation affect generation quality? What experimental evidence isolates the benefit of explicitly modeling this transition?

  2. Could you clarify and reconcile the sources of the baseline results in Table 1? Which REACT edition, task, and official result source does each baseline value correspond to? In particular, why is REGN's FRDist reported as 155.49 rather than the 141.93 reported in its original paper? Were all compared results obtained using the same dataset splits, task settings, and evaluation protocols?

  3. Could you explicitly define gg in Eq. (7)? Please specify its inputs, outputs, and computation, and explain precisely how the Mirror Mapper and FiLM Generator correspond to this mapping.

  4. Does the gate in Eq. (11) receive any sample-specific information beyond the diffusion timestep embedding? If not, how should its role in adapting memory importance to the current context be understood, given that samples at the same diffusion timestep share the same coefficient? Please distinguish the adaptivity provided by the gate ατ\alpha_\tau from that already present in the modulated representation rtr_t.

Reviewer_QFArOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes MNCFR, a framework for generating multiple appropriate listener facial reactions. It introduces a Reaction Formation Module (RFM) to combine speaker behavior with listener-specific information, and Memory-Augmented Regulation (MAR) to improve temporal consistency using recent speaker context. An auxiliary EEG prediction objective is also used to supervise listener-side representations. Experiments demonstrate improvements in reaction appropriateness and temporal smoothness.

Critical strengths

The paper presents a structured framework that separates reaction formation and temporal regulation through RFM and MAR. The proposed method achieves substantial improvements in FRCorr compared to existing methods, demonstrating strong reaction appropriateness. The paper also provides extensive experimental evaluations, including ablation studies, temporal smoothness analyses, and perceptual evaluations, which help demonstrate the effectiveness of the proposed framework.

Critical weaknesses
  1. The paper is difficult to follow, as many sentences contain too much information or are unnecessarily complicated. The overall readability could be improved through clearer explanations and more concise writing.

  2. The paper does not clearly explain why tracking the trajectory of speaker behavior is necessary for the perception-to-response transformation. Consequently, the motivation for using GRU and low-rank recurrent accumulation to capture this trajectory remains unclear.

  3. Although RFM improves generation performance, the individual roles of listener-specific information and current speaker behavior are not sufficiently evaluated. Additional experiments comparing direct feature fusion with the proposed transformation, or independently varying listener history and speaker behavior, would better support the claimed contribution.

  4. EEG is introduced abruptly without defining the acronym or clearly explaining why predicting listener EEG would help learn perception-to-response representations. Although EEG-related experiments are provided, the lack of a clear rationale makes it difficult to interpret whether the results validate the intended role of EEG supervision or simply reflect the benefit of an auxiliary learning objective.

Overall, the paper lacks sufficient motivation for its main contributions and clear rationale for the chosen methodological approaches.

Questions for authors
  1. Could the authors clarify how the transformation gg is implemented? In particular, how are the speaker representation and listener-specific information combined to produce hˉt\bar h_t? Also, why is this considered an explicit perception-to-response transformation, and how does it differ from conventional feature fusion?

  2. Although the paper motivates the use of historical speaker context for temporal continuity, it remains unclear why a separate memory-based regulation pathway is necessary rather than incorporating historical information directly into the existing conditioning process.

  3. In Figure 2, does the encoder in component C receive only the current speaker behavior, or does it also incorporate information from previous windows?

  4. Could the authors clarify the specific implementation of the feature-fusion function ψ\psi in Equation (4)? The paper defines its role but does not clearly describe its architecture.

Reviewer_ThCgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on the Multiple Appropriate Facial Reaction Generation (MAFRG) task, in which the model generates a set of plausible listener behaviors based on the speaker's speech, and the generation process should evolve continuously. Previous work employs an implicit end-to-end framework, while this paper separates current-window reaction conditioning from recent speaker context for explicit organization. The paper conditions the diffusion process with two modules: the Reaction Formation Module encodes the speaker behavior and incorporates the listener reaction history to form a speaker-side representation, with an additional training-time-only EEG head for supervision; the Memory-Augmented Regulation Module further regulates the speaker encoding with a first-in, first-out memory. The results show consistently higher listener facial reaction appropriateness, while the diversity and temporal variation seem to be reduced.

Critical strengths
  • Novel and insightful idea of separating listener history from recent speaker context. The hypothesis is clean and testable, and the authors showed an improvment in the FRCorr metric.
Critical weaknesses
  • Inconsistent improvements over the baseline. Although FRCorr is improved, FRDist is only better in online settings, and the other metrics are consistently worse than the baselines.
  • The EEG prediction head HEEGH_{EEG} is not documented regarding its architecture, number of layers, and formulation, other than a statement that it is a "lightweight prediction head". This limits the reproducibility of the paper.
  • The effect of the auxiliary EEG prediction head is unclear. There isn't an ablation study reporting the metrics when the EEG prediction head is removed. Although Section 4.4 mentions no-EEG controls, the referred figure is unclear, and there isn't a quantitative result comparable to Table 1.
  • There was a required AI use statement to be included for ICLR submission this year, but it appears to be missing from the paper.
Questions for authors
  • As mentioned in the weaknesses, the FRDiv, FRVar, and FRSyn metrics are worse than the baseline. Can the authors explain the implications of this for their submission?
  • The legend of Figure 3 is unclear. In L352-L353, it is claimed that "aligned, shuffled, detached, and no-EEG controls" are evaluated, but there are only 3 experiments named "Mod." "A-EEG" and "S-EEG", without the no-EEG control. Can the author clarify the figure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.