ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation

In dyadic human speaker-listener conversations, the listener's facial reactions play an important role in ensuring that the speaker accurately perceives the listener's emotional states. Since human facial reactions are inherently non-deterministic, the ability to generate multiple appropriate human-like facial reactions is crucial for realistic and engaging human-agent interactions. Although diffusion models are naturally suited to such one-to-many generation, existing diffusion-based Multiple Appropriate Facial Reaction Generation (MAFRG) methods attempt to denoise random Gaussian initialisations directly into multiple appropriate facial reactions (AFRs), conditioned on the perceived speaker behaviour. These random initialisations are usually not well-aligned with the target listener facial reaction, which requires these diffusion models to traverse a complex denoising trajectory from these initialisations, and subsequently creates substantial opportunities for potential deviations away from the range of trajectories leading to appropriate AFRs. Given the inherent mimicry between the human listener's and speaker's facial behaviours, we address the above denoising trajectory issue by leveraging this strong prior. Specifically, we propose ResDiffFRG, a novel diffusion-based MAFRG framework that explicitly anchors the diffusion process to the speaker behaviour by defining its diffusion target as the residual between the speaker anchor and an AFR. Since the speaker anchor already captures substantial target-relevant facial dynamics, the denoiser only needs to model the comparatively small, reaction-specific residual needed to transform this anchor into an AFR, rather than reconstructing the complete reaction from an unstructured state. This requires a simpler denoising trajectory, giving fewer opportunities for potential inappropriate deviations along the trajectory, yielding a simpler generation problem. Extensive experiments on the MARS dataset show that ResDiffFRG achieves large improvements in correlation-based appropriateness over existing methods. We conducted a denoising trajectory analysis, which showed that even at the start of the denoising trajectory, ResDiffFRG already achieves a higher facial-reaction correlation score than the Gaussian Diffusion baseline does after completing 60% of its denoising trajectory. The full training and evaluation code will be released upon acceptance.