acceptodds
Under review as a conference paper at ICLR 2027

AffectAtlas: Structured Neuro-Symbolic Affect Modeling for Emotionally Consistent Talking-Face Generation

Abstract

Audio-driven talking-face generation has made substantial progress in visual fidelity, identity preservation, lip synchronization. However, existing methods often treat emotion as a weak or isolated conditioning signal, leading to facial behaviour that is visually plausible but affectively inconsistent with speech. We propose AffectAtlas, a neuro-symbolic affect modelling framework for emotion-consistent talking-face generation. We model affect as structured system over 3 complementary representations: expressions, action units, valence-arousal. We first construct a probabilistic affect atlas that captures bidirectional relationships among these affect spaces. We then introduce a teacher-guided atlas adaptation mechanism to adapt the atlas to talking-face data while preserving its structured prior. Finally, we incorporate differentiable neuro-symbolic rules to enforce psychologically meaningful consistency among the 3 affect spaces, and inject the refined representations into a diffusion-based generator via an emotion-aware A/V module. Extensive quantitative, qualitative, cross-dataset, ablation and user-study evaluations show that AffectAtlas improves emotion fidelity, temporal affect alignment, visual quality, and overall perceptual realism compared to SOTA methods.

Primary area:
generative models
Keywords:
Affect, Emotion, Talking Face, Facial Action Unit, Valence Arousal, Expression, Neuro-Symbolic

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_cVgMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AffectAtlas, a structured neuro-symbolic framework for emotion-consistent talking-face generation. The method explicitly models bidirectional relationships among three complementary affect spaces: categorical expressions (EXP), facial action units (AUs), and valence-arousal (VA). An offline probabilistic atlas is first constructed and then adapted to the talking-face domain using frozen affect teachers, residual corrections, and cycle consistency. The framework further introduces neuro-symbolic consistency rules and an Audio2Affect student that predicts refined affect representations from speech at inference time. Experiments on multiple talking-face datasets show improvements particularly in emotion-related metrics, while maintaining competitive visual quality and lip synchronization.

Critical strengths
  • The paper addresses an important limitation of prior emotion-aware talking-face methods by explicitly modeling the relationships among complementary EXP, AU, and VA representations rather than treating them as independent conditioning signals.
  • The proposed framework combines probabilistic affect modeling, domain adaptation, and neuro-symbolic constraints in a structured and interpretable manner, providing a conceptually meaningful representation of facial affect.
Critical weaknesses
  • The validity of the proposed structured affect modeling remains insufficiently established. The offline Atlas fits AU-conditioned VA distributions without identifying the source of joint AU-VA annotations. Furthermore, adaptation relies on independently trained teachers whose consistency is not analyzed, while rule satisfaction is evaluated against the same rules enforced during training. These limitations leave the reliability of the learned cross-space relationships insufficiently validated.
  • The evaluation protocol is insufficiently specified to fully assess the reliability of the reported improvements. MEAD and HDTF are used for both training and evaluation, but the paper does not clearly describe the train/test separation or identity overlap. Additionally, EmoNet is used for both VA supervision and evaluation, raising concerns about evaluator dependence.
Questions for authors
  • How are the single-AU and AU-pair-conditioned VA distributions estimated? The paper fits these distributions using GMMs but does not clearly identify datasets with joint AU-VA annotations. Please clarify the source of these statistics and how missing annotations, label differences, and heterogeneous dataset distributions are handled.
  • Are the individual cross-space mappings evaluated independently of the final generation task? In particular, what evidence supports combining EXP, single-AU, and AU-pair VA experts through a Product-of-Experts formulation? Since these experts may contain correlated information, please clarify whether their individual contributions and the resulting mapping accuracy have been evaluated.
  • How consistent are the three frozen teacher streams before atlas adaptation? Since three teachers are independently trained, their predictions may disagree even for the same frame. Is there any quantitative analysis of cross-teacher consistency, and how does the proposed framework handle conflicting supervision across EXP, AU, and VA?
  • How is the coverage and generalizability of the 69 predefined neuro-symbolic rules established? Rule satisfaction is evaluated against the same rules used during training, which does not establish their semantic correctness. Moreover, potentially conflicting cases exist: AU26 is permitted during neutral speech but implies high arousal, whereas neutral expressions imply low arousal. How are such conflicts handled, and is the rule set independently validated across diverse affective conditions?
5 comments

We thank the reviewer for the comments. We address them below; the paper will be updated accordingly.

W1(I) & Q1

Datasets and annotation selection. The following Table1 summarises the datasets and corresponding subsets used to construct the offline AffectAtlas. For Aff-Wild2, we retain only frames with jointly available EXP, AU and VA manual annotations, excluding the Other expression category. For AffectNet, we use approximately 287K manually annotated images with EXP and VA labels, excluding Contempt. For RAF-DB, EmotioNet, DISFA+, and BP4D, we use the corresponding subsets containing both AU annotations and expression information.

Table 1: Overview of affective datasets and their data used for constructing the offline AffectAtlas

Dataset# SamplesAUEXPVA
Aff-Wild2~270K✓✓✓
AffectNet~287K×✓✓
RAF-DB~5K✓✓×
EmotioNet~50K✓✓×
DISFA+~58K✓✓×
BP4D~150K✓✓×

Estimation of AU-conditioned VA distributions. Importantly, both the single-AU and AU-pair VA experts are estimated exclusively from the jointly annotated Aff-Wild2 subset, rather than combining independently annotated AU and VA samples from different datasets. For each AU (k), we collect the frames in which that AU is active and fit a GMM to their corresponding ground-truth VA annotations, obtaining (p(va∣AUk=1))(p(va\mid AU_k=1)). Similarly, for each retained AU pair ((i,j)), we collect frames in which both AUs are simultaneously active and fit a separate GMM to their observed VA annotations, obtaining (p(va∣AUi=1,AUj=1))(p(va\mid AU_i=1,AU_j=1)). Candidate AU pairs are selected based on empirical support, PMI, expression-specific selection, and graph-based pruning to reduce unreliable or redundant combinations.

Missing annotations and dataset heterogeneity. Each conditional distribution is estimated using only the samples containing its required annotations. Specifically, EXP–VA distributions use Aff-Wild2 and AffectNet, EXP–AU relationships use the corresponding EXP/AU-annotated subsets, and AU-conditioned VA experts use only jointly AU/VA-annotated Aff-Wild2 frames. Missing annotations are not imputed or interpreted as negative labels. Expression labels are harmonised to the seven-class taxonomy, with Other and Contempt excluded as appropriate. Consequently, the AU-conditioned VA experts are estimated from a single, consistently annotated source, avoiding the direct pooling of heterogeneous AU and VA annotations from different datasets.

W1(II) & Q3

Having 3 independently trained teachers does not guarantee consistent predictions. However, this is precisely why our atlas adaptation and neuro-symbolic learning mechanisms are interesting and useful.

To evaluate the consistency among the three independently trained affect teachers, we evaluate the agreement between EXP, AU, and VA predictions on 300 randomly selected videos (~45K frames) sampled from all test sets. We compare three configurations: (i) the original frozen teachers, (ii) teacher-guided atlas adaptation without neuro-symbolic constraints, and (iii) the full model.

Cross-space consistency. We measure the agreement between EXP, AU, and VA predictions using three complementary compatibility criteria:

  • EXP–V (Expression–Valence agreement): Measures the proportion of eligible frames in which the predicted valence sign is compatible with the predicted expression. Happiness is typically associated with positive valence, whereas sadness, anger, fear, and disgust are typically associated with negative valence. Surprise and neutral expressions are excluded from this check because they do not imply a consistent positive or negative valence. Its definition is:
\frac{N_{\mathrm{valence\\,consistent}}} {N_{\mathrm{eligible\\,frames}}}$$ - **EXP–A (Expression–Arousal agreement):** Measures the proportion of eligible frames in which the predicted arousal sign is compatible with the predicted expression. Anger, fear, and surprise are treated as typically associated with high arousal, while sadness and neutrality are treated as typically associated with low arousal. Happiness and disgust are excluded because their arousal levels can vary substantially. - **EXP–AU (Expression–Action Unit agreement):** Measures whether the predicted (non-neutral) expression is supported by at least one of its associated AUs, as defined in Table 1(a) of the main paper. An AU is considered active when its predicted probability exceeds 0.5. For EXP–V and EXP–A, VA predictions with absolute magnitude below 0.1 are excluded from the respective agreement calculation, since values near zero do not provide a sufficiently clear positive/negative valence or high/low arousal indication. Such predictions are treated as ambiguous rather than consistent or inconsistent. For EXP–V, the percentages of retained frames are 47.3%, 46.4%, and 47.8% for the original teachers, atlas adaptation, and full model, respectively. For EXP–A, the corresponding percentages are 66.0%, 72.1%, and 70.9%. Each agreement metric is computed on the intersection of frames eligible under all three configurations. The overall conflict rate is defined as the proportion of evaluated frames exhibiting at least one applicable cross-space inconsistency. Lower conflict rates and higher agreement scores indicate greater consistency under these predefined criteria. **Table 1: Cross-space consistency.** | Method | Conflict ↓ | EXP-V ↑ | EXP-A ↑ | EXP-AU ↑ | |---|---:|---:|---:|---:| | Raw teachers | 0.39 | 0.85 | 0.59 | 0.79 | | Atlas adaptation | $\underline{0.28}$ | $\underline{0.89}$ | $\underline{0.71}$ | $\underline{0.82}$ | | **Full model (Atlas adaptation + NeSy rules)** | **0.23** | **0.91** | **0.76** | **0.86** | **As shown in Table 1, the conflict rate decreases from 0.39 for the original teachers to 0.28 after atlas adaptation and further to 0.23 with neuro-symbolic constraints. EXP–V agreement increases from 0.85 to 0.91, while EXP–A agreement increases from 0.59 to 0.76, and EXP–AU agreement increases from 0.79 to 0.86.** **Agreement with an independent pretrained affect estimator.** We additionally compare the predictions against Behaviour4All [1], a separate pretrained facial-behaviour analysis toolkit not used to supervise AffectAtlas. **As shown in Table 2, the complete framework improves agreement relative to the original teachers in EXP Macro-F1 (0.61 to 0.75), AU Macro-F1 (0.49 to 0.56), and valence/arousal CCC (0.65/0.61 to 0.72/0.70).** [1] Kollias, D., Shao, C., Kaloidas, O., & Patras, I.: "Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit", 2024 **Table 2: Agreement with an independently pretrained facial-behaviour estimator (Behaviour4All).** | Method | EXP Macro-F1 ↑ | AU Macro-F1 ↑ | CCCV ↑ | CCCA ↑ | |---|---:|---:|---:|---:| | Raw teachers | 0.61 | 0.49 | 0.65 | 0.61 | | Atlas adaptation | $\underline{0.72}$ | $\underline{0.54}$ | $\underline{0.69}$ | $\underline{0.67}$ | | **Full model (Atlas adaptation + NeSy rules)** | **0.75** | **0.56** | **0.72** | **0.70** | **These findings indicate that atlas adaptation improves cross-space consistency, while the neuro-symbolic constraints provide further improvements. Importantly, the increased agreement with an alternative affect estimator provides complementary evidence that the gains extend beyond the predefined consistency criteria.**

W1(III) & Q4

Rule coverage and independent validation. The 69 neuro-symbolic rules are derived from established facial-behaviour relationships and prior studies (Section 3.3 and Appendix C.3.1). They cover four types of associations: EXP→AU, AU→EXP, EXP→VA and AU→VA, across the seven expression classes. Importantly, these rules represent typical affective associations rather than universally valid constraints, acknowledging that facial movements and affective states can vary across individuals and contexts.

To investigate their effectiveness beyond rule satisfaction, we conducted additional experiments on 300 randomly selected videos across the test sets, comparing (i) the original frozen affect teachers, (ii) Atlas adaptation without neuro-symbolic constraints, and (iii) the full framework.

Cross-space consistency. We evaluate EXP–V, EXP–A, and EXP–AU agreement, together with the overall proportion of frames exhibiting at least one applicable cross-space conflict. As shown in Table 1, Atlas adaptation reduces the conflict rate from 0.39 to 0.28, while incorporating the neuro-symbolic rules further reduces it to 0.23 (a 17.9% relative reduction compared with Atlas adaptation alone). The full framework also improves EXP–V agreement from 0.89 to 0.91, EXP–A agreement from 0.71 to 0.76, and EXP–AU agreement from 0.82 to 0.86. These results demonstrate that the neuro-symbolic constraints provide additional cross-space consistency beyond the probabilistic Atlas.

Evaluation against an independent affect estimator. Since these consistency measures reflect predefined affective relationships, we additionally evaluate agreement with Behaviour4All [1], a separately pretrained facial-behaviour analysis model that was not used to supervise AffectAtlas. Compared with Atlas adaptation without neuro-symbolic constraints, the full framework improves EXP Macro-F1 (0.72→0.75), AU Macro-F1 (0.54→0.56), CCC-V (0.69→0.72), and CCC-A (0.67→0.70), as shown in Table 2 of our previous response. These findings provide complementary evidence that the neuro-symbolic constraints improve agreement with an external affect estimator, rather than merely increasing satisfaction of the rules used during training. While cross-model agreement does not constitute ground-truth validation of every individual rule, it supports the effectiveness of the proposed constraints beyond the original teachers and predefined consistency metrics.

[1] Kollias, D., Shao, C., Kaloidas, O., & Patras, I.: "Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit", 2024

Handling conflicting rules. AU26 (jaw drop) may occur during neutral speech due to articulation and does not necessarily indicate high emotional arousal. Our framework accommodates such ambiguity through soft, differentially weighted constraints, rather than enforcing rules as hard implications. Specifically, EXP→VA rules receive a weight of 1.0, whereas AU→VA rules receive a lower weight of 0.35, reflecting the fact that individual AU activations may also arise from non-emotional facial movements. Consequently, when Neutral and AU26 are simultaneously activated with comparable confidence, Neutral→LowA contributes more strongly than AU26→HighA. This weighting does not eliminate the competing preferences; instead, it allows them to be balanced during optimisation. During randomized source-space training, one of the three frozen teacher outputs (EXP, AU, or VA) is selected as the source, while the other two representations are predicted through the corresponding Atlas mappings and residual adapters. The neuro-symbolic loss is applied to the resulting affect triplet, while the target-space teacher outputs provide supervision through the alignment losses. Furthermore, the neuro-symbolic objective is combined with teacher-alignment and cycle-consistency losses, and its overall weight is gradually increased to 0.5, preventing the symbolic constraints from dominating teacher-guided adaptation from the outset. Thus, the resulting arousal prediction is determined jointly by the weighted rule penalties and teacher supervision, rather than being dictated by either rule in isolation.

Generalisability across datasets and independent VA evaluation. We further investigate whether the proposed rule-constrained framework produces meaningful affective dynamics beyond the EmoNet teacher used during training. To this end, we independently re-evaluate the generated and reference videos using EmotiEffLib, a separately pretrained VA estimator not used during AffectAtlas training, and recompute CCC-V, CCC-A, and temporal response lag for all competing methods under the same evaluation protocol.

Reviewer_esMKOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AffectAtlas, a neuro-symbolic framework, is designed for emotionally consistent, audio-driven talking-face generation. Instead of treating emotions as isolated labels, the authors unify three complementary affect spaces (EXP, AUs, VA) into a structured probabilistic prior. To align this offline prior with talking-face video data, they introduce a teacher-guided adaptation mechanism alongside a neuro-symbolic consistency framework that enforces psychologically grounded facial relationships through differentiable fuzzy logic rules. During inference, an Audio2Affect (student) network predicts these consistent affect tokens directly from speech audio to drive a diffusion-based video generator, achieving realistic emotional fidelity, temporal alignment, and visual realism compared to state-of-the-art baselines.

Critical strengths

** The manuscript is clear and easy to follow. Talking face generation is a broad research area with numerous early and recent studies, and the same applies to emotion-related works. Despite the limited space, the authors have done a good job of balancing the coverage of these topics, summarizing the literature, and organizing the models and advancements in the Related Work section. However, I believe the review would not be complete without citing at least the following two papers, as they also focus on generating emotionally consistent talking faces. PS: Due to the reference style, I may have overlooked these papers. If they are already cited, please disregard this comment. [1] Ji, X., Zhou, H., Wang, K., Wu, Q., Wu, W., Xu, F., & Cao, X. (2022, July). EAMM: One-shot emotional talking face via audio-based emotion-aware motion model. In ACM SIGGRAPH 2022 Conference Proceedings (pp. 1-10). [2] Köksal, A., Xu, Q., & Lim, J. H. (2024, June). Talking face generation via face mesh controllability without reference videos. In 2024 IEEE Conference on Artificial Intelligence (CAI) (pp. 1380-1386). IEEE.

** The authors provide comprehensive quantitative and qualitative evaluations across multiple benchmarks (MEAD, HDTF, RAVDESS, and VFHQ), comparing against 8 models using 13 evaluation metrics.

Critical weaknesses

** The framework relies heavily on multiple pre-trained models. These components increase the overall computational time. While fidelity, correctness, smoothness … are important for talking face generation, inference speed is also a critical consideration. However, the paper does not discuss computational efficiency or elapsed time, nor does it provide any experiments comparing the video generation speed of the proposed method against competing approaches. (Including any latency experiment or at least a discussion about it would improve the overall quality of the work as such an analysis could help address questions regarding the real-world applicability of the proposed model.

Questions for authors

NA

1 comment

We thank the reviewer for the comment. We address it below; the paper will be updated accordingly.

W1

We clarify that the three pretrained visual affect teachers are used exclusively during training and are not required at inference. Instead, the Audio2Affect module directly predicts affect tokens from speech, which are then used to condition the video generator. Consequently, AffectAtlas requires only a single reference image and speech audio at inference, without additional visual affect estimation, manually specified emotion labels, or motion-driving references.

To quantitatively assess the computational requirements of AffectAtlas, we benchmark it against five diffusion-based and three non-diffusion-based talking-face generation methods on an NVIDIA A100 40GB GPU. We evaluate the generation of 10-second videos, using a standardised output frame rate of 25 FPS and a resolution of 512×512 for diffusion-based methods. We report total generation time, inference throughput (generated frames per second), peak GPU memory consumption, and model parameters. Parameter counts include all neural networks required by each method's evaluated inference pipeline, excluding training-only networks, discriminators, and optimiser states.

As shown in the Table, AffectAtlas generates a 10-second video in 270.3 seconds, achieving 0.92 inference FPS with 18.61 GB peak GPU memory. Its generation time is comparable to MEMO (260.1 s), while being lower than EchoMimic (313.0 s) and Hallo (405.6 s). Although DiceTalk and KeyFace achieve faster inference, AffectAtlas requires less GPU memory than both methods (18.61 GB versus 20.30 GB and 29.94 GB, respectively), as well as MEMO (19.30 GB). Non-diffusion-based approaches achieve substantially faster generation and lower memory consumption, although they are evaluated at a lower output resolution of 256×256. We include these results as additional reference points rather than directly equivalent computational comparisons. Furthermore, certain competing methods employ additional emotion, expression, or pose conditioning for controllable expressive generation, whereas AffectAtlas directly derives its affective conditioning from speech.

Table: Comparison of inference efficiency across SOTA methods.

Diffusion-based methods

MethodOutputGeneration Time (s) ↓Inference FPS ↑Peak GPU (GB) ↓Params (M)
DiceTalk512×512, 25 fps186.5‾\underline{186.5}1.34‾\underline{1.34}20.302548.8
KeyFace512×512, 25 fps158.61.5829.943558.2
MEMO512×512, 25 fps260.10.9619.303051.6
EchoMimic512×512, 25 fps313.00.808.402242.9
Hallo512×512, 25 fps405.60.6210.19‾\underline{10.19}2726.4
Ours512×512, 25 fps270.30.9218.613080.7

Non-diffusion-based methods (reference)

MethodOutputGeneration Time (s) ↓Inference FPS ↑Peak GPU (GB) ↓Params (M)
EDTalk256×256, 25 fps11.322.123.5967.1
SadTalker256×256, 25 fps13.3‾\underline{13.3}18.80‾\underline{18.80}5.14‾\underline{5.14}240.9
EAT256×256, 25 fps22.611.065.64398.6
Reviewer_SLXTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

AffectAtlas addresses emotion-consistent audio-driven talking-face generation by jointly modeling categorical expressions, facial action units, and valence–arousal. It constructs probabilistic mappings between these spaces, adapts them using frozen visual affect teachers, and applies differentiable logical constraints to encourage consistency. The resulting affect tokens condition a diffusion generator. At inference, an Audio2Affect student predicts these tokens from speech.

Critical strengths
  • The representation addresses a meaningful limitation of single-space emotion conditioning.
  • The emotion-related improvements extend across datasets. The reported RAVDESS and VFHQ evaluations support transfer beyond the datasets used for generator training.
Critical weaknesses
  • The atlas construction remains insufficiently specified at important interfaces. Although several affect datasets are listed, the paper does not provide a dataset-level account of observed versus imputed labels, missing-label handling, or sample counts for these conditional models.
  • The quantitative VA evidence and some reported comparisons need stronger validation. EmoNet supplies VA teacher supervision and also extracts the VA trajectories used for CCC and response-lag evaluation. These measurements establish agreement with that estimator, but provide limited independent validation of frame-level affect. The human studies help support perceptual quality, although they do not directly validate VA trajectories.
Questions for authors

The VA and response-lag improvements need validation using an independent estimator or human annotations.

4 comments

We thank the reviewer for the comments. We address them below; the paper will be updated accordingly.

W1

The following Table summarises the datasets and corresponding samples used to construct the offline AffectAtlas. For Aff-Wild2, we retain only the frames with jointly available EXP, AU and VA manual annotations, excluding samples labelled as Other. For AffectNet, we use the manually annotated subset containing EXP and VA labels, excluding the Contempt. For RAF-DB, we select the subset containing both EXP and AU annotations. Similarly, for EmotioNet, we use the manually annotated subset with EXP and AU labels. For DISFA+ and BP4D, we retain the subsets with both AU annotations and corresponding EXP information. These datasets provide complementary supervision for estimating the conditional distributions of AffectAtlas. Specifically, EXP–VA relationships are estimated using Aff-Wild2 and AffectNet, while EXP–AU relationships are estimated using Aff-Wild2, RAF-DB, EmotioNet, DISFA+ and BP4D. The AU-conditioned VA distributions, including single-AU and AU-pair experts, are estimated using the jointly annotated Aff-Wild2 samples. Each conditional distribution is therefore constructed using the available annotations relevant to that relationship, without requiring all three affect spaces to be annotated for every sample.

Table 1:Overview of affective datasets and their data used for constructing the offline AffectAtlas

Dataset# SamplesAUEXPVA
Aff-Wild2~270K✓✓✓
AffectNet~287K×✓✓
RAF-DB~5K✓✓×
EmotioNet~50K✓✓×
DISFA+~58K✓✓×
BP4D~150K✓✓×

W2 & Q

To address this concern, we conduct an additional evaluation using EmotiEffLib (model: enet_b0_8_va_mtl), an independently trained VA estimator that is not used during AffectAtlas training. Specifically, we re-extract the frame-level valence and arousal trajectories from both the generated and ground-truth videos and recompute CCC-V and CCC-A. Importantly, we re-evaluate all eight competing methods using the same estimator and evaluation protocol, ensuring a consistent comparison. As shown in Table 2, AffectAtlas achieves the highest CCC-V and CCC-A on all four datasets under EmotiEffLib. Specifically, it obtains 0.76/0.76 on MEAD, 0.78/0.76 on RAVDESS, 0.69/0.66 on HDTF, and 0.65/0.61 on VFHQ. These results consistently outperform the strongest competing methods and corroborate our original findings under EmoNet.

We additionally recompute response delay using the independently extracted VA trajectories. As shown in Table 3, AffectAtlas obtains an average absolute lag of 5.23, 7.89, 3.91, and 5.64 frames on MEAD, RAVDESS, HDTF, and VFHQ, respectively. These measurements provide an additional assessment of temporal affect alignment that does not depend on the EmoNet predictions used for teacher supervision. Overall, the results demonstrate that the observed improvements in VA consistency are not specific to the EmoNet evaluator, but remain evident under an independently trained affect estimation model.

Table 2: Comparison of VA evaluation using EmotiEffLib and EmoNet.

DatasetCCCV (EmotiEffLib) ↑CCCA (EmotiEffLib) ↑CCCV (EmoNet) ↑CCCA (EmoNet) ↑
MEAD0.760.760.720.70
RAVDESS0.780.760.720.73
HDTF0.690.660.610.56
VFHQ0.650.610.590.61

 

Table 3: Comparison of VA evaluation with SOTA using EmotiEffLib.

MethodMEAD CCCV ↑MEAD CCCA ↑RAVDESS CCCV ↑RAVDESS CCCA ↑HDTF CCCV ↑HDTF CCCA ↑VFHQ CCCV ↑VFHQ CCCA ↑
SadTalker0.540.470.640.600.460.470.540.52
EAT0.580.530.680.620.480.460.520.45
EDTalk0.540.560.73‾\underline{0.73}0.670.440.460.460.48
MEMO0.67‾\underline{0.67}0.650.710.69‾\underline{0.69}0.460.450.550.51
Hallo0.540.560.610.570.510.560.550.53
DiceTalk0.610.66‾\underline{0.66}0.720.660.510.540.61‾\underline{0.61}0.49
EchoMimic0.560.530.640.590.460.460.540.54
KeyFace0.630.630.700.630.63‾\underline{0.63}0.60‾\underline{0.60}0.600.55‾\underline{0.55}
Ours0.760.760.780.760.690.660.650.61

 

Table 4: Comparison of response delay using EmotiEffLib and EmoNet (frames/seconds; lower is better).

DatasetEmotiEffLib (Independent estimator) ↓EmoNet (Original estimator) ↓
MEAD5.23 / 0.21s6.02 / 0.24s
RAVDESS7.89 / 0.32s7.17 / 0.28s
HDTF3.91 / 0.16s6.29 / 0.25s
VFHQ5.64 / 0.23s6.69 / 0.27s

 

EmotiEffLib: https://github.com/sb-ai-lab/EmotiEffLib

Table4: Comparison of response delay with SOTA using EmotiEffLib across four datasets (frames/seconds; lower is better).

MethodMEAD ↓RAVDESS ↓HDTF ↓VFHQ ↓
SadTalker8.81 / 0.35s10.39 / 0.42s4.81 / 0.19s7.33 / 0.29s
EAT9.11 / 0.36s8.02/0.32s‾\underline{8.02 / 0.32\mathrm{s}}5.12 / 0.20s6.87 / 0.27s
EDTalk8.20 / 0.33s8.52 / 0.34s4.17 / 0.17s8.12 / 0.32s
MEMO10.93 / 0.36s9.60 / 0.32s6.14 / 0.20s8.24 / 0.27s
Hallo7.35 / 0.29s8.92 / 0.36s4.00 / 0.16s6.21 / 0.25s
DiceTalk6.39/0.26s‾\underline{6.39 / 0.26\mathrm{s}}8.52 / 0.34s2.91 / 0.12s6.02/0.24s‾\underline{6.02 / 0.24\mathrm{s}}
EchoMimic8.08 / 0.32s10.02 / 0.40s4.01 / 0.16s6.30 / 0.25s
KeyFace8.93 / 0.36s10.60 / 0.42s4.75 / 0.19s8.47 / 0.34s
Ours5.23 / 0.21s7.89 / 0.32s3.91/0.16s‾\underline{3.91 / 0.16\mathrm{s}}5.64 / 0.23s

We thank the reviewer again for their constructive feedback.

In our previous response, we provided further clarifications regarding the datasets, annotation protocols, and construction of the offline AffectAtlas. We also conducted extensive additional evaluations using EmotiEffLib, an independent affect estimator, and re-evaluated all competing methods under the same protocol. These results further support our original findings across all four datasets and demonstrate that the observed improvements are not specific to the original EmoNet evaluator.

We hope that these clarifications and additional experimental results have addressed the reviewer's concerns.

Thanks for the rebuttal on the evaluation experiments, which addresses some of my concerns. I'll raise my rating to 3.

Reviewer_gYigOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uses three descriptions of facial affect—expression categories (EXP), facial action units (AU), and valence–arousal (VA)—to guide talking-face generation. It first builds probabilistic mappings between these descriptions using affect datasets. It then adapts the mappings to talking-face videos using pretrained visual models and adds 69 predefined rules to discourage conflicting predictions. An audio student learns to predict the resulting affect features, which are used to condition a diffusion generator. At test time, the method requires a reference portrait and driving audio. The experiments include comparisons with existing methods, component ablations, cross-dataset tests, and human evaluations.

Critical strengths
  1. The method connects useful descriptions of facial behavior. Expression categories, local facial actions, and VA describe different aspects of a face’s expression. Modeling their relationships is a reasonable way to guide generation. The audio student also allows the method to use the learned visual information without requiring a target video at test time.

  2. The paper includes useful controls and human evaluations. The MLP and Transformer baselines in Tables 3 and 17 use the same three affect signals, making them relevant comparisons. Table 14 also compares the full method with versions that replace the atlas or remove the rules. The reported human ratings support the benefits of both components, and the cross-dataset tests provide further evidence that the method works beyond its training data.

Critical weaknesses
  1. The paper needs to show why this particular atlas design is needed. Earlier work already uses EXP–AU relationships to guide learning [1, 2]. This paper extends the modeling to VA and uses the resulting representation for talking-face generation. The ablations in Tables 3–5, 14, and 17 provide useful evidence for the atlas, the combination of affect signals, and the logic loss. However, the paper does not report separate ablations for the MRF interaction terms or the AU-pair VA experts, leaving their individual contributions unclear. Comparing versions that remove each component separately, while keeping the data and other training settings unchanged, would help show what these design choices add.

  2. The use of the rules during neutral speech needs clarification Appendix C.3.1 allows AU26 (jaw drop) during neutral speech, while Tables 10–11 include both Neutral → LowA and AU26 → HighA. If both rules use the same arousal estimate, they favor opposite values when the Neutral and AU26 predictions are both high. The soft loss and teacher supervision can balance these terms, so this does not by itself show incorrect behavior. However, the paper does not clearly specify which predictions each rule uses during source-space training or how the rule weights are chosen. Table 14 supports the overall benefit of NeSy, but a clarification of these details would help explain its behavior in this case.

[1] Cui et al. Knowledge Augmented Deep Neural Networks for Joint Facial Expression and Action Unit Recognition. NeurIPS, 2020.

[2] Kollias et al. Distribution Matching for Multi-Task Learning of Classification Tasks: A Large-Scale Study on Faces & Beyond. AAAI, 2024.

Questions for authors
  1. How do the results change when the MRF interaction terms and, separately, the AU-pair VA experts are removed, with the other settings unchanged?

  2. Appendix C.3.1 allows AU26 during neutral speech, while Tables 10–11 include both Neutral → LowA and AU26 → HighA. These rules push arousal in opposite directions when both predictions are high. How are their weights chosen, and how are these penalties balanced against teacher supervision in such cases?

  3. With the generator fixed, how do the results differ between using visual-teacher tokens and audio-predicted tokens? This would help show how much performance changes when the learned visual affect representation is predicted from audio.

4 comments

We thank the reviewer for the comments. We address them below; the paper will be updated accordingly.

W1 & Q1

To isolate the contributions of these two key components of the AffectAtlas, we conduct two additional controlled ablations on RAVDESS:

(i) Without MRF interactions: We remove the pairwise interaction terms from the expression-conditioned AU model, thereby assuming conditional independence among AUs given the expression, while retaining the individual EXP–AU probabilities.

(ii) Without AU-pair VA experts: We remove the AU-pair VA experts from the Product-of-Experts formulation, retaining the EXP-level and single-AU VA experts, as well as the complete MRF.

All other components and experimental settings are kept unchanged. We evaluate the models on 150 RAVDESS test videos, with results presented in the table below.

Table: Ablation of MRF interactions and AU-pair VA experts on RAVDESS.

ModalityEFID ↓EmoAcc ↑CCCV (EmotiEffLib) ↑CCCA ( EmotiEffLib )↑CCCV ( EmoNet )↑CCCA ( EmoNet)↑F1 ↑
wo MRF0.53‾\underline{0.53}50.30.610.600.580.540.59
wo AU-pair VA experts0.5555.7‾\underline{55.7}0.67‾\underline{0.67 }$\underline{ 0.62 } $$\underline{0.62 } $0.59‾\underline{ 0.59}0.66‾\underline{0.66}
Ours0.4069.710.780.760.720.730.73

Note: We evaluated CCC-V and CCC-A using both EmoNet (the original VA estimator used in the main paper) and an independent valence–arousal (VA) estimator from EmotiEffLib (model: enet_b0_8_va_mtl). The independent estimator was included to avoid relying on the same model for both training and evaluation.

Contribution of MRF interactions: Removing the pairwise MRF interactions substantially degrades performance, reducing AU F1 from 0.73 to 0.59 and EmoAcc from 69.71% to 50.30%. This demonstrates the importance of modelling expression-dependent AU co-activation patterns rather than treating AUs as conditionally independent given the expression. The reductions in VA CCC further suggest that modelling coordinated facial actions benefits the overall structured affect representation.

Contribution of AU-pair VA experts: Removing the AU-pair VA experts also leads to consistent performance degradation, reducing CCC-V from 0.78 to 0.67, CCC-A from 0.76 to 0.62, and EmoAcc from 69.71% to 55.70%. These results support the contribution of pairwise AU-conditioned affect information beyond the EXP-level and individual-AU VA experts.

Overall, both ablations consistently underperform the complete model across all five metrics. The findings indicate that the two components contribute complementary relational information: the MRF captures dependencies among facial action activations conditioned on expressions, whereas the AU-pair VA experts capture the affective information associated with their co-activation. Both contribute to improved emotion fidelity in the generated videos.

EmotiEffLib: https://github.com/sb-ai-lab/EmotiEffLib

W2 & Q2

AU26 (jaw drop) can occur during neutral speech due to articulation and does not necessarily indicate high emotional arousal. Our neuro-symbolic framework handles such potentially conflicting relationships through soft, differentially weighted constraints, rather than enforcing them as hard rules. Specifically, EXP-to-VA rules are assigned a weight of 1.0, whereas AU-to-VA rules receive a lower weight of 0.35, reflecting the fact that individual AU activations may also arise from non-emotional facial movements. Consequently, when Neutral and AU26 are both strongly activated, the Neutral → LowA rule contributes more strongly than the AU26 → HighA rule, favouring lower arousal while retaining the latter as a weaker constraint. Importantly, this weighting does not eliminate the competing preferences but allows them to be reconciled during optimisation. During randomized source-space training, one of the three frozen teacher outputs (EXP, AU, or VA) is selected as the source, while the other two representations are predicted through the corresponding Atlas mappings and residual adapters. The neuro-symbolic objective is combined with cross-teacher alignment and cycle-consistency losses. The overall logic-loss weight is gradually increased to 0.5, preventing the symbolic constraints from dominating teacher-guided adaptation from the outset. Thus, the resulting arousal prediction is determined jointly by the weighted rule penalties and teacher supervision, rather than by either rule in isolation.

Q3

To quantify the effect of cross-modal affect distillation, we compare generation conditioned on visual-teacher tokens extracted from the reference ground-truth videos with generation conditioned on audio-predicted student tokens, keeping the generator and all other inference settings fixed. As shown in the table, visual-teacher conditioning improves EmoAcc from 69.71% to 73.12%, valence/arousal CCC from 0.78/0.76 to 0.80/0.78, and AU F1 from 0.73 to 0.76. This indicates that the audio student retains much of the visual teacher's affective information, with modest reductions in emotion-related accuracy. Conversely, the student achieves slightly better FVD, illustrating that improved recognition-oriented affect metrics do not necessarily translate into improved distribution-level video metrics.

Table :Comparison of visual-teacher and audio-predicted affect tokens on RAVDESS.

ModalityFVD ↓EFID ↓EmoAcc ↑CCCV (EmotiEffLib) ↑CCCA (EmotiEffLib) ↑CCCV (EmoNet) ↑CCCA (EmoNet) ↑F1 ↑
Ours with visual teacher token179.110.3973.120.800.780.760.740.76
Ours with student affect token174.030.4069.710.780.760.720.730.73

Thank you for the ablation and clarifications. Additional experiments address my main concerns, and I would raise my rating to 3.

Thank you for your careful consideration and updated assessment. We sincerely appreciate your constructive feedback, which helps us further improve the quality of the final manuscript.

We noticed that the numerical rating still appears unchanged in the review form. We would be grateful if you could update it to reflect your revised assessment.

Thank you again for your feedback and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.