- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AffectAtlas, a structured neuro-symbolic framework for emotion-consistent talking-face generation. The method explicitly models bidirectional relationships among three complementary affect spaces: categorical expressions (EXP), facial action units (AUs), and valence-arousal (VA). An offline probabilistic atlas is first constructed and then adapted to the talking-face domain using frozen affect teachers, residual corrections, and cycle consistency. The framework further introduces neuro-symbolic consistency rules and an Audio2Affect student that predicts refined affect representations from speech at inference time. Experiments on multiple talking-face datasets show improvements particularly in emotion-related metrics, while maintaining competitive visual quality and lip synchronization.
Critical strengths
- The paper addresses an important limitation of prior emotion-aware talking-face methods by explicitly modeling the relationships among complementary EXP, AU, and VA representations rather than treating them as independent conditioning signals.
- The proposed framework combines probabilistic affect modeling, domain adaptation, and neuro-symbolic constraints in a structured and interpretable manner, providing a conceptually meaningful representation of facial affect.
Critical weaknesses
- The validity of the proposed structured affect modeling remains insufficiently established. The offline Atlas fits AU-conditioned VA distributions without identifying the source of joint AU-VA annotations. Furthermore, adaptation relies on independently trained teachers whose consistency is not analyzed, while rule satisfaction is evaluated against the same rules enforced during training. These limitations leave the reliability of the learned cross-space relationships insufficiently validated.
- The evaluation protocol is insufficiently specified to fully assess the reliability of the reported improvements. MEAD and HDTF are used for both training and evaluation, but the paper does not clearly describe the train/test separation or identity overlap. Additionally, EmoNet is used for both VA supervision and evaluation, raising concerns about evaluator dependence.
Questions for authors
- How are the single-AU and AU-pair-conditioned VA distributions estimated? The paper fits these distributions using GMMs but does not clearly identify datasets with joint AU-VA annotations. Please clarify the source of these statistics and how missing annotations, label differences, and heterogeneous dataset distributions are handled.
- Are the individual cross-space mappings evaluated independently of the final generation task? In particular, what evidence supports combining EXP, single-AU, and AU-pair VA experts through a Product-of-Experts formulation? Since these experts may contain correlated information, please clarify whether their individual contributions and the resulting mapping accuracy have been evaluated.
- How consistent are the three frozen teacher streams before atlas adaptation? Since three teachers are independently trained, their predictions may disagree even for the same frame. Is there any quantitative analysis of cross-teacher consistency, and how does the proposed framework handle conflicting supervision across EXP, AU, and VA?
- How is the coverage and generalizability of the 69 predefined neuro-symbolic rules established? Rule satisfaction is evaluated against the same rules used during training, which does not establish their semantic correctness. Moreover, potentially conflicting cases exist: AU26 is permitted during neutral speech but implies high arousal, whereas neutral expressions imply low arousal. How are such conflicts handled, and is the rule set independently validated across diverse affective conditions?
5 comments
W1(II) & Q3
Having 3 independently trained teachers does not guarantee consistent predictions. However, this is precisely why our atlas adaptation and neuro-symbolic learning mechanisms are interesting and useful.
To evaluate the consistency among the three independently trained affect teachers, we evaluate the agreement between EXP, AU, and VA predictions on 300 randomly selected videos (~45K frames) sampled from all test sets. We compare three configurations: (i) the original frozen teachers, (ii) teacher-guided atlas adaptation without neuro-symbolic constraints, and (iii) the full model.
Cross-space consistency. We measure the agreement between EXP, AU, and VA predictions using three complementary compatibility criteria:
- EXP–V (Expression–Valence agreement): Measures the proportion of eligible frames in which the predicted valence sign is compatible with the predicted expression. Happiness is typically associated with positive valence, whereas sadness, anger, fear, and disgust are typically associated with negative valence. Surprise and neutral expressions are excluded from this check because they do not imply a consistent positive or negative valence. Its definition is:
\frac{N_{\mathrm{valence\\,consistent}}} {N_{\mathrm{eligible\\,frames}}}$$ - **EXP–A (Expression–Arousal agreement):** Measures the proportion of eligible frames in which the predicted arousal sign is compatible with the predicted expression. Anger, fear, and surprise are treated as typically associated with high arousal, while sadness and neutrality are treated as typically associated with low arousal. Happiness and disgust are excluded because their arousal levels can vary substantially. - **EXP–AU (Expression–Action Unit agreement):** Measures whether the predicted (non-neutral) expression is supported by at least one of its associated AUs, as defined in Table 1(a) of the main paper. An AU is considered active when its predicted probability exceeds 0.5. For EXP–V and EXP–A, VA predictions with absolute magnitude below 0.1 are excluded from the respective agreement calculation, since values near zero do not provide a sufficiently clear positive/negative valence or high/low arousal indication. Such predictions are treated as ambiguous rather than consistent or inconsistent. For EXP–V, the percentages of retained frames are 47.3%, 46.4%, and 47.8% for the original teachers, atlas adaptation, and full model, respectively. For EXP–A, the corresponding percentages are 66.0%, 72.1%, and 70.9%. Each agreement metric is computed on the intersection of frames eligible under all three configurations. The overall conflict rate is defined as the proportion of evaluated frames exhibiting at least one applicable cross-space inconsistency. Lower conflict rates and higher agreement scores indicate greater consistency under these predefined criteria. **Table 1: Cross-space consistency.** | Method | Conflict ↓ | EXP-V ↑ | EXP-A ↑ | EXP-AU ↑ | |---|---:|---:|---:|---:| | Raw teachers | 0.39 | 0.85 | 0.59 | 0.79 | | Atlas adaptation | $\underline{0.28}$ | $\underline{0.89}$ | $\underline{0.71}$ | $\underline{0.82}$ | | **Full model (Atlas adaptation + NeSy rules)** | **0.23** | **0.91** | **0.76** | **0.86** | **As shown in Table 1, the conflict rate decreases from 0.39 for the original teachers to 0.28 after atlas adaptation and further to 0.23 with neuro-symbolic constraints. EXP–V agreement increases from 0.85 to 0.91, while EXP–A agreement increases from 0.59 to 0.76, and EXP–AU agreement increases from 0.79 to 0.86.** **Agreement with an independent pretrained affect estimator.** We additionally compare the predictions against Behaviour4All [1], a separate pretrained facial-behaviour analysis toolkit not used to supervise AffectAtlas. **As shown in Table 2, the complete framework improves agreement relative to the original teachers in EXP Macro-F1 (0.61 to 0.75), AU Macro-F1 (0.49 to 0.56), and valence/arousal CCC (0.65/0.61 to 0.72/0.70).** [1] Kollias, D., Shao, C., Kaloidas, O., & Patras, I.: "Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit", 2024 **Table 2: Agreement with an independently pretrained facial-behaviour estimator (Behaviour4All).** | Method | EXP Macro-F1 ↑ | AU Macro-F1 ↑ | CCCV ↑ | CCCA ↑ | |---|---:|---:|---:|---:| | Raw teachers | 0.61 | 0.49 | 0.65 | 0.61 | | Atlas adaptation | $\underline{0.72}$ | $\underline{0.54}$ | $\underline{0.69}$ | $\underline{0.67}$ | | **Full model (Atlas adaptation + NeSy rules)** | **0.75** | **0.56** | **0.72** | **0.70** | **These findings indicate that atlas adaptation improves cross-space consistency, while the neuro-symbolic constraints provide further improvements. Importantly, the increased agreement with an alternative affect estimator provides complementary evidence that the gains extend beyond the predefined consistency criteria.**
W1(III) & Q4
Rule coverage and independent validation. The 69 neuro-symbolic rules are derived from established facial-behaviour relationships and prior studies (Section 3.3 and Appendix C.3.1). They cover four types of associations: EXP→AU, AU→EXP, EXP→VA and AU→VA, across the seven expression classes. Importantly, these rules represent typical affective associations rather than universally valid constraints, acknowledging that facial movements and affective states can vary across individuals and contexts.
To investigate their effectiveness beyond rule satisfaction, we conducted additional experiments on 300 randomly selected videos across the test sets, comparing (i) the original frozen affect teachers, (ii) Atlas adaptation without neuro-symbolic constraints, and (iii) the full framework.
Cross-space consistency. We evaluate EXP–V, EXP–A, and EXP–AU agreement, together with the overall proportion of frames exhibiting at least one applicable cross-space conflict. As shown in Table 1, Atlas adaptation reduces the conflict rate from 0.39 to 0.28, while incorporating the neuro-symbolic rules further reduces it to 0.23 (a 17.9% relative reduction compared with Atlas adaptation alone). The full framework also improves EXP–V agreement from 0.89 to 0.91, EXP–A agreement from 0.71 to 0.76, and EXP–AU agreement from 0.82 to 0.86. These results demonstrate that the neuro-symbolic constraints provide additional cross-space consistency beyond the probabilistic Atlas.
Evaluation against an independent affect estimator. Since these consistency measures reflect predefined affective relationships, we additionally evaluate agreement with Behaviour4All [1], a separately pretrained facial-behaviour analysis model that was not used to supervise AffectAtlas. Compared with Atlas adaptation without neuro-symbolic constraints, the full framework improves EXP Macro-F1 (0.72→0.75), AU Macro-F1 (0.54→0.56), CCC-V (0.69→0.72), and CCC-A (0.67→0.70), as shown in Table 2 of our previous response. These findings provide complementary evidence that the neuro-symbolic constraints improve agreement with an external affect estimator, rather than merely increasing satisfaction of the rules used during training. While cross-model agreement does not constitute ground-truth validation of every individual rule, it supports the effectiveness of the proposed constraints beyond the original teachers and predefined consistency metrics.
[1] Kollias, D., Shao, C., Kaloidas, O., & Patras, I.: "Behaviour4All: in-the-wild Facial Behaviour Analysis Toolkit", 2024
Handling conflicting rules. AU26 (jaw drop) may occur during neutral speech due to articulation and does not necessarily indicate high emotional arousal. Our framework accommodates such ambiguity through soft, differentially weighted constraints, rather than enforcing rules as hard implications. Specifically, EXP→VA rules receive a weight of 1.0, whereas AU→VA rules receive a lower weight of 0.35, reflecting the fact that individual AU activations may also arise from non-emotional facial movements. Consequently, when Neutral and AU26 are simultaneously activated with comparable confidence, Neutral→LowA contributes more strongly than AU26→HighA. This weighting does not eliminate the competing preferences; instead, it allows them to be balanced during optimisation. During randomized source-space training, one of the three frozen teacher outputs (EXP, AU, or VA) is selected as the source, while the other two representations are predicted through the corresponding Atlas mappings and residual adapters. The neuro-symbolic loss is applied to the resulting affect triplet, while the target-space teacher outputs provide supervision through the alignment losses. Furthermore, the neuro-symbolic objective is combined with teacher-alignment and cycle-consistency losses, and its overall weight is gradually increased to 0.5, preventing the symbolic constraints from dominating teacher-guided adaptation from the outset. Thus, the resulting arousal prediction is determined jointly by the weighted rule penalties and teacher supervision, rather than being dictated by either rule in isolation.
Generalisability across datasets and independent VA evaluation. We further investigate whether the proposed rule-constrained framework produces meaningful affective dynamics beyond the EmoNet teacher used during training. To this end, we independently re-evaluate the generated and reference videos using EmotiEffLib, a separately pretrained VA estimator not used during AffectAtlas training, and recompute CCC-V, CCC-A, and temporal response lag for all competing methods under the same evaluation protocol.
We thank the reviewer for the comments. We address them below; the paper will be updated accordingly.
W1(I) & Q1
Datasets and annotation selection. The following Table1 summarises the datasets and corresponding subsets used to construct the offline AffectAtlas. For Aff-Wild2, we retain only frames with jointly available EXP, AU and VA manual annotations, excluding the Other expression category. For AffectNet, we use approximately 287K manually annotated images with EXP and VA labels, excluding Contempt. For RAF-DB, EmotioNet, DISFA+, and BP4D, we use the corresponding subsets containing both AU annotations and expression information.
Table 1: Overview of affective datasets and their data used for constructing the offline AffectAtlas
Estimation of AU-conditioned VA distributions. Importantly, both the single-AU and AU-pair VA experts are estimated exclusively from the jointly annotated Aff-Wild2 subset, rather than combining independently annotated AU and VA samples from different datasets. For each AU (k), we collect the frames in which that AU is active and fit a GMM to their corresponding ground-truth VA annotations, obtaining (p(va∣AUk=1)). Similarly, for each retained AU pair ((i,j)), we collect frames in which both AUs are simultaneously active and fit a separate GMM to their observed VA annotations, obtaining (p(va∣AUi=1,AUj=1)). Candidate AU pairs are selected based on empirical support, PMI, expression-specific selection, and graph-based pruning to reduce unreliable or redundant combinations.
Missing annotations and dataset heterogeneity. Each conditional distribution is estimated using only the samples containing its required annotations. Specifically, EXP–VA distributions use Aff-Wild2 and AffectNet, EXP–AU relationships use the corresponding EXP/AU-annotated subsets, and AU-conditioned VA experts use only jointly AU/VA-annotated Aff-Wild2 frames. Missing annotations are not imputed or interpreted as negative labels. Expression labels are harmonised to the seven-class taxonomy, with Other and Contempt excluded as appropriate. Consequently, the AU-conditioned VA experts are estimated from a single, consistently annotated source, avoiding the direct pooling of heterogeneous AU and VA annotations from different datasets.