acceptodds
Withdrawn from ICLR 2027

Beyond End-to-End Black Box Mapping: An Intentional Agent Framework for Cognitive-driven Facial Reaction Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Automatic human-like facial reaction generation (FRG) is essential for building intelligent systems that can authentically and deeply engage in human-computer interaction (HCI). While diverse and context-appropriate facial reactions can reflect latent appraisal and affective processes in human interaction, most existing FRG methods rely on end-to-end architectures that directly map speaker behaviours to listener expressions without an explicit intermediate internal state. In this paper, we reformulate FRG as generation mediated by a structured internal-state process and propose the Intentional Agent, which shifts FRG from direct stimulus-response mapping to stimulus-grounded generation through explicit intermediate states. To represent temporal internal-state evolution for FRG, we propose an internal dynamics model that integrates emotional drives with an iterative Inner Thought Flow (ITF) within a structured intermediate state used for subsequent generation. This state can continue to update during conversational silences. Furthermore, to bridge abstract internal states with physiological actions, we explicitly formulate FRG as a downstream affective mapping from this latent thought flow to physical facial expressions. Experiments on the REACT 2025 dataset show an FRDist of 72.39 and an FRDiv of 0.5057; perceptual plausibility is evaluated separately through blinded human ratings. A blinded human evaluation of 96 reactions found no significant difference in mean score between Full and ground truth ( vs. , ), while Full significantly outperformed Event-Triggered and Heuristic-Only (both ). The Reaction Quality Scorer (RQS) correlated strongly with human judgements (Pearson ; Spearman , both ), supporting its use as a supplementary automatic metric. These results underscore the immense potential of endogenous dynamics in building highly autonomous, human-like agents.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Intentional Agents, Cognitive Modeling, Facial Reaction Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_anHGOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Intentional Agent framework to address the lack of explicit internal-state modelling and continued state evolution during conversational silence in Facial Reaction Generation. Specifically, the framework uses Inner Thought Flow to integrate memory, affect, thought, and expectation, enabling the listener‘s internal state to update even when no new external event arrives. It further combines a VAD-to-Action Unit mapping with a Diffusion Transformer to translate internal affective states into facial reactions and generate fine-grained facial dynamics. Experiments and human evaluation on REACT 2025, together with adaptation experiments on Seamless Interaction, demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. This paper formulates facial reaction generation as an explicit internal-state-mediated process and considers state evolution during conversational silence. This provides a valuable modelling perspective for listener reactions with history dependence and temporal continuity.
  2. The evaluation includes standard facial reaction metrics, blinded human ratings, and internal-state ablations. This paper also validates the agreement between its Reaction Quality Scorer and human judgements.
Critical weaknesses
  1. The key comparison changes both ITF updates and the update strategy during silence, making it difficult to determine whether the improvements arise from explicit ITF or simply from continued state updates. Moreover, the internal-state ablations primarily assess action-trigger rate, expectation error, and memory retention, without sufficiently establishing how these changes affect the quality of the generated facial reactions. Consequently, the necessity and additional benefits of ITF over simpler alternatives remain unclear.
  2. The method achieves favourable FRDist and diversity scores, but its FRCorr is substantially lower than that of several baselines, and its FRSyn is not the best. The human evaluation does not include competitive existing MAFRG methods, leaving it unclear whether these metric trade-offs correspond to more natural and context-appropriate facial reactions. Although reliance on automatic metrics is common in this field, the pronounced disagreement among the reported metrics warrants further investigation.
Questions for authors
  1. Under the same update frequency and a comparable computational budget, would replacing ITF with a history summary or state updates without thought text significantly reduce performance? Could such controls distinguish the contribution of explicit inner thoughts from that of continued updates?
  2. Given that Full has a substantially lower FRCorr than several baselines, how do the authors determine whether this reflects valid one-to-many reactions rather than insufficient contextual alignment? Could blinded human comparisons with representative competitive baselines clarify the trade-offs among the metrics?
  3. How is "zero external input" defined in the silence experiments? Are visual signals still provided when the speaker is silent? If so, how are the effects of internal-state evolution distinguished from those of continued visual input?
  4. How do errors in the text-to-VAD mapping affect the generated facial reactions? Given the relatively weak correlations with human annotations for Arousal and Dominance, could the authors assess this interface by replacing the affect estimator or using human-annotated VAD values?
Reviewer_G4kKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript reformulates MAFRG by replacing direct, end-to-end stimulus-to-reaction black-box mappings with an intentional agent framework that mediates facial expression generation through an explicit, self-updating internal state. The main contribution is an internal dynamics model, driven by a zero-shot Qwen3-8B LLM that maintains memory, affect, inner thought flow, and predictive expectations, allowing state updates and proactive action preparation even during conversational silences without external triggers. The system projects these latent cognitive states onto physical facial expressions by combining hand-crafted VAD-to-FACS action unit mapping rules for low-frequency macroscopic signals with a diffusion transformer for high-frequency residual dynamics. The authors validate their framework on the REACT 2025 benchmark, demonstrating superior FRDist (72.39) and FRDiv (0.5057) over baseline methods, and conduct a blinded human study where their full architecture achieves perceptual plausibility comparable to ground-truth video recordingsand also a custom, human-validated RQS.

Critical strengths

The conceptual motivation, qualitative figures, and overall framing grounded in cognitive appraisal theory are well-articulated. The paper successfully addresses a key limitation in existing FRG models to maintain realistic, evolving non-verbal reactions during prolonged pauses without external speaker triggers. By introducing a heartbeat mechanism and structured state maintenance (memory, expectation, emotion), the system generates continuous, contextually coherent facial micro-expressions during silences, as shown by improvements over event-triggered baselines and high human ratings for temporal continuity. The authors provide an evaluation combining standard trajectory metrics on REACT 2025, recording-group-disjoint split generalization tests, cross-benchmark adaptation on seamless interaction, detailed controlled diversity/drift audits, and a blinded human study with high inter-rater reliability. This empirical structure effectively substantiates their qualitative and perceptual claims beyond simple frame-level metric comparisons.

Critical weaknesses

Eqs. 1-7 relies on pseudo-formal notation where mathematical operators represent prompt-engineering heuristics rather than rigorous or unified probabilistic/algebraic formulations. Readers must frequently navigate between the main text and extensive appendices to discern exact operational implementations. The translation from text/thought to visual expressions relies on a complex, multi-stage pipeline involving noisy text-to-VAD lexicon lookup, rigid piecewise FACS mapping rules, 1D Gaussian smoothing, and explicit synthetic sine-wave breathing/swaying overlays. Consequently, it remains unclear whether the high human evaluation scores stem from the core LLM-driven intentional state evolution or simply from the visual polish provided by hand-crafted temporal smoothing and breathing heuristics applied to stylized 3D avatars. On standard REACT 2025 metrics, the model's FRCorr drops to 0.08 indicating near-zero frame-aligned correlation with actual human listener trajectories, while FRDiv (0.5057) exceeds gt (0.1876), indicating potential over-generation of motion range. Furthermore, the end-to-end update latency of 4.37 seconds is incompatible with real-time backchanneling and interactive conversational interfaces where human listener reactions occur within sub-second windows. Many pre-print references. Some incomplete references.

Questions for authors
  1. Could the authors provide human evaluation results for an ablation condition that removes the hand-crafted low-frequency components to isolate how much visual plausibility is driven by these graphic heuristics versus the LLM's inner thought flow?
  2. How does the system handle real-time turn-taking or rapid backchanneling when a speaker speaks continuously? Is the 15-sec silence heartbeat interval sufficient to maintain conversational responsiveness in live HCI deployments?
  3. Why does FRCorr drop so severely to 0.08 compared to existing MAFRG baselines (0.54–0.71)? Does this reflect a fundamental trade-off where the model prioritizes internal cognitive drift over temporal alignment with the speaker's vocal cues?
Reviewer_D5AkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies facial reaction generation (FAR) that generates facial response during conversation. To address the limitation of prior work that directly maps speaker behaviors to listener expressions without considering intermediate internal state modeling, the authors propose Intentional Agent, an agentic approach that utilizes explicit internal state. Specifically, the paper first proposes an internal dynamics model that uses iterative Inner Thought Flow (ITF) for emotion drive. Then the agentic framework responds with current perception, memory, and thought. In the experiment, the propose approach outperforms the baselines for three out of five quantitative metrics on its main benchmark. Furthermore, it introduces Reaction Quality Scorer (RQS) as a supplementary metric that is aligned with human evaluation.

Critical strengths
  • The idea of explicitly modeling internal state for facial reaction generation seems interesting and new.
Critical weaknesses
  • The motivation of using agentic approach for facial reaction generation is not well-motivated. In facial reaction generation, current end-to-end mapping approaches may be still effective for the purpose of the task and why we need more complicate and expensive agentic approach is needed is unclear, e.g., why do we need a memory for this task as directly generating facial response may be still enough rather than retrieving past experience. Similarly, leveraging longer context or internal emotion drive and state might not be necessary for generating listener's facial expression, which may be usually instant reaction to the speaker.
  • The experimental evaluation does not demonstrate the effectiveness of the approach. First, in its main result in Table 1, the proposed approach still underperforms the baselines in two out of five metrics. Therefore, human evaluation is essential but there is no user study for baseline comparison. In addition, since it uses LLM, cost comparison with the baselines is also needed to report like latency as fairer comparison.
  • Writing needs improvement. First, the method section has many detail but is sometimes hard to follow, e.g., Sec. 3.2. Figure 1 and 2 can be also improved for better clarity. Second, it also lacks several important details, e.g., the detail for the proposed Reaction Quality Scorer (RQS) is missing, such as its motivation and how to compute and train RQS. Furthermore, it was not clearly stated how Event-Triggered and Heuristic-Only are performed.
Questions for authors
  • How is Reaction Quality Scorer (RQS) computed and trained? What is the motivation of it?
  • Can you provide cost comparison with the baselines like latency?
  • Can you provide detail of Event-Triggered and Heuristic-Only approaches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.