acceptodds
Withdrawn from ICLR 2027

ResDiffFRG: Residual Diffusion for Multiple Appropriate Facial Reaction Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In dyadic human speaker-listener conversations, the listener's facial reactions play an important role in ensuring that the speaker accurately perceives the listener's emotional states. Since human facial reactions are inherently non-deterministic, the ability to generate multiple appropriate human-like facial reactions is crucial for realistic and engaging human-agent interactions. Although diffusion models are naturally suited to such one-to-many generation, existing diffusion-based Multiple Appropriate Facial Reaction Generation (MAFRG) methods attempt to denoise random Gaussian initialisations directly into multiple appropriate facial reactions (AFRs), conditioned on the perceived speaker behaviour. These random initialisations are usually not well-aligned with the target listener facial reaction, which requires these diffusion models to traverse a complex denoising trajectory from these initialisations, and subsequently creates substantial opportunities for potential deviations away from the range of trajectories leading to appropriate AFRs. Given the inherent mimicry between the human listener's and speaker's facial behaviours, we address the above denoising trajectory issue by leveraging this strong prior. Specifically, we propose ResDiffFRG, a novel diffusion-based MAFRG framework that explicitly anchors the diffusion process to the speaker behaviour by defining its diffusion target as the residual between the speaker anchor and an AFR. Since the speaker anchor already captures substantial target-relevant facial dynamics, the denoiser only needs to model the comparatively small, reaction-specific residual needed to transform this anchor into an AFR, rather than reconstructing the complete reaction from an unstructured state. This requires a simpler denoising trajectory, giving fewer opportunities for potential inappropriate deviations along the trajectory, yielding a simpler generation problem. Extensive experiments on the MARS dataset show that ResDiffFRG achieves large improvements in correlation-based appropriateness over existing methods. We conducted a denoising trajectory analysis, which showed that even at the start of the denoising trajectory, ResDiffFRG already achieves a higher facial-reaction correlation score than the Gaussian Diffusion baseline does after completing 60% of its denoising trajectory. The full training and evaluation code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Models, Multiple Appropriate Facial Reaction Generation, One-to-many Mapping

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p8hgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies multiple appropriate facial reaction generation, where several plausible listener facial reactions are generated from a speaker's audio-visual behaviour. Existing diffusion methods denoise directly from Gaussian noise to the listener reaction, which the authors argue starts too far from the target. Because listeners tend to mimic speakers, the paper instead diffuses the difference between the listener and speaker emotion representations. At inference the residual is added back to the speaker representation and passed through a fixed feasibility projection. The authors also standardize each residual dimension with training-set statistics so that all dimensions share the same signal-to-noise ratio at each timestep. Experiments on the MARS dataset with the REACT 2025 splits compare against Trans-VAE, REGNN and PerFRDiff, and include a controlled ablation of the diffusion target and an analysis of the denoising trajectory.

Critical strengths

The controlled ablation is clean. Table 2 changes only the diffusion target while keeping the architecture, noise schedule and conditioning fixed, and FRCorr rises from 0.69 to 0.80, a gain that can be attributed fairly clearly to the residual target. The trajectory analysis in Table 4 also shows that residual diffusion already reaches at its first step the level the baseline needs about sixty percent of the denoising steps to reach, which is consistent with the authors' motivation

Critical weaknesses

The first problem is limited novelty. Diffusing the residual between a conditioning anchor and the target is well established in image restoration and generation, for example ResShift and ResDiff for super-resolution, as well as RDDM, InDI, I2SB and bridge diffusion models in general. None of this work is cited or discussed, and the claim of being the first to formulate the task this way holds only within the narrow scope of facial reaction generation. Per-dimension standardization is routine preprocessing. Overall the paper applies an existing technique to a specific task, and the methodological contribution for the ICLR audience is weak.

Second, the results do not convincingly show that the method is better. The improvement is almost entirely in FRCorr. On FRDist the two variants score 180.7 and 199.0, both worse than PerFRDiff at 177.8 and clearly worse than REGNN at 141.9, and the normalized variant has an FRDiv of only 0.097 compared with 0.139 for PerFRDiff. More importantly, the B_Mime baseline, which simply copies the speaker, already reaches an FRCorr of 0.52, close to the 0.56 of PerFRDiff. FRCorr thus rewards outputs that stay close to the speaker, and it is hard to tell whether the model learns appropriate reactions or small perturbations of the speaker's expressions. In addition, only one dataset is used, no variance over runs is reported, there is no comparison with recent diffusion methods such as ReactDiff, SC-Diff, VQ-Diff and SAFRG, and there are no rendered results or user studies. Three different models in Tables 2 and 3 have exactly the same FRSyn of 45.73, which the authors should check.

Questions for authors
  1. How far are the generated reactions from the speaker representation, and how does that compare with the distance between real listeners and the speaker? If the baseline is the speaker representation plus the mean residual or plus random noise, does ResDiffFRG win on all metrics?
  2. Why do the normalized variant, its version without CFG and the Stochastic Baseline have identical FRSyn values?
  3. Could you report the mean and standard deviation over several random seeds, and compare with ReactDiff, SC-Diff and VQ-Diff under the REACT 2025 protocol?
  4. Apart from the choice of anchor, what is the essential technical difference between this work and ResShift, RDDM, InDI or bridge diffusion?
Reviewer_q8xAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ResDiffFRG for offline multiple appropriate facial reaction generation. Rather than directly predict the behavior of listener, the method models the residual between speaker and listener facial behaviour representations using conditional diffusion and reconstructs listener reactions by adding generated residuals to the speaker representation. A normalised variant standardises residual dimensions using training-set statistics to address scale differences. Experiments on the MARS dataset show improved correlation-based appropriateness over the evaluated baselines, with trade-offs in distance-based appropriateness and diversity.

Critical strengths
  1. The idea of generating the difference between speaker and listener expressions is intuitive and fits the task, given the tendency of listeners to mirror speakers’ facial behaviour.
  2. The controlled comparison in Table 2 is useful: changing the target to a residual improves FRCorr from 0.69 to 0.80, with little change in inter-sample diversity, while keeping the architecture, conditioning inputs, and loss fixed.
Critical weaknesses
  1. Limited novelty. The method mainly changes the diffusion target from listener features to speaker–listener residuals, and adds per-dimension normalisation (which is a commonly used trick). The denoiser architecture and sampling procedure remain largely unchanged. The ablations support the usefulness of this change, but the methodological contribution is relatively incremental. Moreover, the performance in Table 2 does not show impressive advance over other methods. Although ResDiffFRG achieves higher FRCorr, both variants have worse FRDist than PerFRDiff, and the normalised variant also has lower diversity (FRDiv). The improvements are therefore not consistent across metrics.
  2. The paper claims that learning residuals makes denoising easier, but the experiments do not demonstrate this. Section 4.1 describes the residual distribution as less complex, although subtracting the speaker representation does not necessarily make the conditional distribution simpler. In Section 6, the higher initial correlation in Table 4 can be explained by adding the speaker anchor to the intermediate outputs. This shows the benefit of the anchor, but does not establish that the denoising trajectory is simpler. Faster convergence or comparable quality with fewer sampling steps would provide more convincing evidence.
  3. The normalised variant also changes the loss weights from (1,5,5) to (1,1,1), making it difficult to separate the effects of normalisation and loss reweighting. Applying the same normalisation to the direct-generation baseline would also help clarify the source of the gains.
  4. Lack of video demos. For facial reaction generation, static frames are insufficient to assess temporal smoothness and whether the reactions are appropriate to the speaker’s behaviour.
Questions for authors
  1. Does the normalised variant still improve performance when the loss weights are kept the same as those of the direct residual variant?
  2. Could the authors provide video comparisons with the baselines to assess motion smoothness and the appropriateness of the reactions?
Reviewer_t5VWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ResDiffFRG generates listener facial reactions by modeling the residual between the listener's facial attribute representation and the speaker's representation. At inference, the generated residual is added to the speaker anchor and passed through a feasibility projection. A normalized variant standardizes residual dimensions using training statistics, and the model uses a Transformer denoiser with classifier free guidance. Experiments on the official MARS splits compare the method with published baselines and a matched diffusion model that predicts the full listener representation. The direct residual formulation raises FRCorr from 0.69 to 0.80 relative to that matched model. Normalization improves FRDist but reduces diversity. A trajectory analysis shows higher initial correlation after reconstruction around the speaker anchor.

Critical strengths
  1. The underlying idea is intuitive and well-motivated, as leveraging the speaker's facial behavior as a prior naturally aligns with the characteristics of the task.

  2. The matched target ablation provides useful evidence for the speaker anchor. Table 2 compares full target diffusion and direct residual diffusion using the same denoiser architecture, conditioning signals, and training losses. With guidance, FRCorr improves from 0.69 to 0.80 while FRDiv remains almost unchanged, and the correlation advantage also appears without guidance. This supports a real benefit of the residual parameterization within the tested setup. The formulation is straightforward to implement and uses a meaningful prior available at inference, making it a useful task specific finding.

Critical weaknesses
  1. Insufficient evidence of overall improvement. Although the proposed method improves facial reaction correlation, it performs worse on distributional quality or diversity compared to existing baselines. Since generating diverse and appropriate reactions is the central objective, correlation alone is insufficient to establish superiority. The lack of human evaluation, multiple datasets, and comparisons with recent methods further weakens the experimental evidence.

  2. Insufficient validation of the proposed mechanism. Residual modeling may provide a useful inductive bias, but the paper does not convincingly demonstrate that it simplifies the denoising process. The trajectory analysis does not establish reduced computational complexity or fewer generation failures. Additionally, the normalization experiments change both the target representation and loss weights, making it difficult to isolate the contribution of residual normalization.

Questions for authors
  1. Evidence request: Does the residual formulation improve perceived reaction appropriateness and maintain meaningful diversity when compared with the matched full target model and stronger recent baselines? What are the uncertainty estimates for FRCorr, FRDist, and FRDiv, and do the improvements persist across different training seeds?

  2. Evidence request: What happens when residual normalization and group loss weights are varied independently? Does a full target model with comparable normalization, loss weighting, and output projection recover the normalized model's FRDist improvement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.