acceptodds
Under review as a conference paper at ICLR 2027

MAM-REACTOR: ANCHOR-PRESERVING SET-LEVEL ALIGNMENT FOR MULTIPLE APPROPRIATE FACIAL REACTION GENERATION

Abstract

Offline Multiple Appropriate Facial Reaction Generation (MAFRG) requires more than producing multiple plausible appropriate facial reactions (AFRs): the generated reactions should be both individually appropriate and collectively diverse. Existing offline MAFRG approaches mainly address one-to-many reaction generation through distribution-level modeling or instance-wise supervision. The former models each reaction sequence holistically, capturing global distributional trends but providing weaker fine-grained frame-level alignment, whereas the latter directly supervises individual reactions but does not coordinate multiple predictions as a set. Consequently, neither formulation explicitly organizes the finite output set to complementarily cover multiple valid reactions for the same interaction. We therefore formulate offline MAFRG as a set-level alignment problem. To this end, we propose Mam-Reactor, which combines trajectory-level anchoring with set-level alignment. A deterministic paired-reaction anchor provides a reliable fine-grained appropriateness reference, while expression-identified residual queries and diversity regulation expand it into an expressive candidate set in one forward pass. Balanced prediction–target set alignment then jointly optimizes the correspondence between the complete prediction set and multiple valid same-session reactions, encouraging different candidates to cover complementary valid reaction modes rather than redundantly favoring the same target. A lightweight isolated VA refinement further improves continuous affective trajectories without altering the learned set structure. Experiments on the offline MARS benchmark show that our Mam-Reactor achieved an FRC of 1.0888 and an FRD of 134.81 at an FRDiv of 0.1521. It further generates all ten reactions in a single forward pass, providing approximately faster inference than the 50-step diffusion baseline.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multiple Appropriate Facial Reaction Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_roRFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MAM-Reactor generates ten facial reactions using a frozen deterministic anchor and nine query-conditioned residual candidates. It uses a bidirectional Mamba backbone, expression-specific queries, diversity constraints, balanced Sinkhorn alignment to a set of same-session reactions, and a final restricted valence–arousal refinement. The method targets the quality–diversity trade-off in multiple appropriate facial reaction generation.

Critical strengths

Preserving an anchor while learning alternatives is a reasonable approach to this trade-off. The staged experiments show how diversity and reaction distance change as components are added. The paper reports both target-to-prediction and Hungarian coverage diagnostics, rather than relying only on prediction-to-target distances. Producing the set in one forward pass is practically attractive. The supplement includes model code and a checkpoint.

Critical weaknesses

1. The experiments do not establish why balanced transport is the right set-alignment mechanism. The central idea uses established ingredients: parallel learned queries, entropy-regularized transport, and a sequence-alignment loss. These need to be positioned against work on set prediction, Sinkhorn transport and Soft-DTW [1–3]; the Mamba backbone also needs its original citation [4]. These are methodological references, not additional facial-reaction baselines.

More substantially, the current ablation compares the set-alignment stage with removing the alignment package. It does not compare balanced transport against nearest-target matching, a bidirectional/Chamfer objective, or hard Hungarian matching using the same costs, regularizers and training budget. P2 to P3 also adds training. Without these controls, the gains could come from the added style/FRD supervision rather than balancing the correspondence. Balanced marginals constrain transport mass, but do not guarantee distinct outputs: identical candidates can split their mass over several targets.

There is also a modeling assumption to test. Uniform weighting treats the available same-session reactions as equally representative, and eight fixed expression queries encourage particular expression types irrespective of their appropriateness in a given context. The compatibility head and mixture query help, but do not by themselves establish that all eight explicit candidates are valid alternatives. Please measure per-query appropriateness and compare uniform with context-dependent target weighting.

2. The evidence remains too dependent on one benchmark's scoring geometry. Several objectives directly reproduce aspects of FRD and FRDiv, including AU rounding, channel allocation and a diversity budget. Optimizing these quantities is legitimate, but an independent evaluation is needed to show that the resulting diversity corresponds to plausible reactions. The reported coverage gains are useful, yet remain distances to the same finite session-level target set. There is no human evaluation or second dataset establishing that the additional candidates are natural and contextually appropriate, and no variability across training seeds or sampled target sets.

The synchrony evaluation is especially limited. Appendix A.1 computes FRSyn using only the first candidate, which is the frozen anchor; it therefore cannot assess the nine learned alternatives. The reported 49-frame value is also the boundary of the lag search, while ground truth is already close to that boundary. This raises a question about the diagnostic, rather than proving that the generated motion is delayed by exactly 49 frames. Please validate it and report synchrony for the complete set. The supplementary README further identifies the checkpoint as evaluated against session-all-GT targets, while training samples ten targets. This can be a valid protocol, but the exact target pool and evaluator version must be specified for every comparison, including imported baseline numbers. The archive deliberately omits the evaluator and training launchers, so the provided checkpoint alone does not make the reported table reproducible.

I lean toward rejection because the central alignment choice and the validity of the generated alternatives need stronger evidence.

Questions for authors
  1. How does balanced OT compare with nearest-target, bidirectional/Chamfer and Hungarian matching under identical costs and training budgets?
  2. Are all explicit expression queries appropriate for each input? Please report per-query failures and test target weighting rather than assuming uniform mass.
  3. Can independent raters judge the complete reaction set, and do the conclusions survive different random seeds and target-set samples?
  4. Can you clarify FRSyn's use of the anchor only, its frequent boundary optimum, and the exact all-GT evaluation protocol? Please provide the evaluator/configuration needed to reproduce the checkpoint results.

Presentation

The main text repeats the anchor/diversity/alignment motivation, while many decisions that actually determine the objective appear only in the appendix. A compact loss/stage table would be more useful. Please fix references such as “Eq. equation 31” and add the missing methodological citations. These writing issues are secondary to the experimental concerns above.

References

[1] Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances. https://arxiv.org/html/1306.0895

[2] Carion et al. End-to-End Object Detection with Transformers. Sec. 3.1, parallel query-based set prediction and bipartite matching. https://arxiv.org/html/2005.12872

[3] Cuturi and Blondel. Soft-DTW: a Differentiable Loss Function for Time-Series. https://arxiv.org/html/1703.01541

[4] Gu and Dao. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. https://arxiv.org/html/2312.00752

Reviewer_YxTdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper works on Multiple Appropriate Facial Reaction Generation(MAFRG), which aims to generate appropriate facial reactions (AFRs) conditioned on the observed speaker behavior. Existing approaches therefore organize MAFRG supervision mainly either at the level of a reaction distribution or at the level of an individual prediction-target relation. The proposed method explores to preserves the temporally aligned paired-AFR reference and coordinates the complete prediction set toward complementary valid target.

The method first learns a deterministic anchor from paired facial reactions to provide a reference for generation. It then expands this anchor into a diverse set of facial reaction candidates using expression-identified residual queries and diversity regularization. Experiments on the MARS dataset demonstrate that they can achieve a trade-off between reaction appropriateness and diversity.

Critical strengths
  1. The proposed anchor-based generation strategy is intuitive and reasonable.
  2. The experimental results demonstrate that the proposed method achieves a great trade-off between facial reaction appropriateness and diversity.
Critical weaknesses
  1. The presentation could be further improved. For example, some abbreviations are used without being explicitly defined in the main text, such as "VA" in Line 30 and "AU" in Line 226.
  2. The use of a deterministic one-to-one mapping to construct the anchor requires further justification. Given the inherently one-to-many nature of the task, similar speaker behaviors may elicit different appropriate listener reactions. A single deterministic anchor may therefore introduce an unnecessary constraint on the diversity of generated reactions.
  3. The qualitative results in Figure 3 do not clearly demonstrate the advantages of the proposed method over existing approaches. Only with the input image sequences and GT listener reactions, it is difficult to determine which method generates more appropriate facial reactions.
  4. According to Table 1, the proposed method achieves the worst synchrony performance among the compared methods. A more detailed analysis is needed to explain this performance degradation.
Questions for authors
  1. From Table 1, the proposed method achieves great improvements in reaction appropriateness, but seemingly at the cost of reduced diversity. Can appropriateness and diversity be explicitly balanced during inference?
  2. According to the description in the paper, Instance-wise methods like PerFRDiff learn from direct paired reactions, which should perform great on reaction appropriateness while perform worse on diversity. However, in table 1, PerFRDiff can achieve similar diversity with the proposed method with a much worse appropriateness. Can authors explain why?
Reviewer_eVcSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles offline Multiple Appropriate Facial Reaction Generation (MAFRG). The authors argue that existing methods organize supervision at either distribution-level (one shared conditional distribution) or instance-level (each prediction supervised independently) Neither matches the actual evaluation protocol, which requires a finite set of candidates that collectively cover multiple valid reactions. Hence, they propose Mam-Reactor, which formulates offline MAFRG as a set-level alignment problem.

Critical strengths

The reframing of MAFRG as finite set-level supervision is a well-motivated formulation. The official metrics (FRC sums per-candidate best-match scores; FRD sums per-candidate min distances) are inherently set-structured, and neither distribution-level nor instance-level training directly optimizes the joint coverage structure. The paper articulates this gap convincingly.

Critical weaknesses
  1. The diversity numbers (FRVar) don't match the claims, which needs direct addressing. The headline diversity claim is FRDiv 0.1521, near Huang et al.'s 0.1510. But FRVar is 0.0601 vs. Huang et al.'s 0.0801 and MReactor's 0.1507, which shows a 25–60% drop. The paper never discusses FRVar in the main text beyond Table 1. Given the paper's own framing ("individually appropriate and collectively diverse"), the diversity claim rests on FRDiv alone, which measures cross-candidate distance.
  2. How are the 10 valid targets obtained, and does K = Q matter? MARS has 137 listeners over 2,856 session pairs, so same-session alternatives per speaker clip should be plentiful, but the target-set construction procedure (deterministic epoch-dependent shuffle, repeated placeholders with zero mass) deserves a robustness check: do results hold with K ∈ {5, 10, 15} or random re-sampling? If Sinkhorn is balanced with uniform marginals and K_i < Q, what happens?
  3. A single dataset limits the generality of "set-level alignment" as a claim. If any second dataset or an adapted protocol exists (e.g., using REACT 2023/2024 data in offline form), even one additional validation would help.
  4. Facial reaction generation for digital humans has misuse potential (synthetic listener behavior in persuasion/manipulation contexts). The ethics statement is mostly about data usage. A short discussion of intended use and limitations would be appropriate.
Questions for authors

Please refer to weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.