acceptodds
Withdrawn from ICLR 2027

LEMO: LLM-Guided Fine-Grained Emotion Editing for Speech-Driven Facial Motion Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Although speech-driven facial motion generation has advanced, faithfully synthesizing expressions that match emotional prosody remains a challenging task. This is mainly due to the close coupling of emotional cues and phonetic content within audio signals, which results in learned representations where these aspects are entangled. From our analysis, we observe two prevalent failure modes in current models: Expression Attenuation, where facial animations lack the intended emotional intensity even when the audio input is expressive, and Emotion Mismatch, where the model reproduces emotions from phonetically similar training samples instead of the user's desired emotion. Rather than addressing these limitations solely during generation, we propose LEMO, a motion-level emotion editing framework. LEMO decomposes facial motion into upper-face movement, phonetically driven neutral lip motion, and expressive lip offsets, using a phone-driven decomposition approach. To support flexible, text-based editing, expressive offsets are discretized as structured region-state descriptors, achieved by spatial and temporal quantization of blendshape activation. This tokenized interface enables a large language model to interpret natural language instructions and deliver region- and time-specific expression changes. A diffusion-based temporal inpainter, together with a region-aware ControlNet, then applies these modifications additively, preserving lip synchronization. Quantitative experiments on the MEAD dataset show that LEMO sets a new benchmark for both expression fidelity and motion naturalness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D facial animation, Speech-driven facial motion generation, Emotion editing, Large language model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3YJbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LEMO, a motion-level emotion editing framework for speech-driven 3D facial animation motivated by two failure modes: Expression Attenuation and Emotion Mismatch. LEMO decomposes facial motion into upper-area motion, phone-driven neutral lip motion and expressive lip offsets. Through spatial and temporal quantization of blendshape activations it constructs structured region-state descriptors that enable an LLM to interpret free-form text instructions and generate region- and time-specific edits. A diffusion-based temporal inpainter coupled with a region-aware ControlNet applies these edits while keeping neutral lip motion unmasked to preserve lip synchronization. Experiments on MEAD and MEAD-OpenVoice evaluate high-level emotion editing while VLM-based evaluation assesses fine-grained instruction alignment. The authors report improvements in emotion editing and competitive lip synchronization compared with existing methods.

Critical strengths

Relevant formulation of fine-grained expression editing: The paper addresses a practically useful problem: enabling “region-specific or temporally localized edits” in speech-driven facial animation (Section 1). Its motion-level editing formulation allows users to specify facial actions beyond global emotion labels. The “please blink twice” example in Fig. 3 illustrates the intended control over a particular facial region and action count which is valuable for interactive animation editing.

Explicit separation of editable and preserved motion components: The motion inpainting formulation distinguishes upper-area motion and expressive lip offsets from phone-driven neutral lip motion (Sections 4.1–4.2). In particular neutral lip motion is “always kept unmasked” (L207) providing a speech-related motion reference while the model edits expressive components. This is a useful design choice for conditioning expression editing on the underlying articulation.

Promising quantitative results for emotion editing and lip synchronization: Table 1 reports the highest F-mAP and lowest LSE-D among the compared methods on both MEAD and MEAD-OpenVoice. On MEAD, LEMO achieves an F-mAP of 0.398 compared with 0.364 for the strongest baseline on this metric and an LSE-D of 9.83 compared with the best baseline value of 10.90. Table 2 also reports a higher VLM-based instruction alignment score than ExpCLIP (4.02 vs. 3.78). These results provide empirical support for the framework’s editing capabilities while maintaining competitive lip synchronization.

Critical weaknesses

Insufficient validation of the pseudo-neutral decomposition: Section 3.2 describes pseudo-neutral speech as preserving “phone content while removing expressive prosodic cues” (L149) and interprets the residual as “emotion-related deviations.” (L153) However the ASR–TTS–animation pipeline can introduce pronunciation differences and motion generation errors. Motion-Aligned processing aligns phone timing but does not necessarily remove these differences Appendix A.2 also acknowledges “potential interpolation artifacts.” (L672-673) Consequently the residual may contain non-emotional discrepancies in addition to expressive motion. The authors should quantify residual errors on neutral utterances and assess sensitivity to the TTS, animation generator and alignment procedure.

Limited validation of the VLM-based comparative evaluation: Section 5.1 states that Qwen3.7-Plus is used “as the LLM and VLM” (L323) but no evaluation with an alternative VLM is reported. Appendix D.2 provides a useful human-agreement check, yet it covers only 20 edited samples from “our method results only” (L982) with a Spearman correlation of 0.614 (Table 9). This does not establish whether the VLM’s ranking of LEMO against ExpCLIP agrees with human judgment.

Ambiguous training protocol for compositional generalization: Section 5.2 defines an unseen combination as one that “never appears in any training descriptor string” (L428) but subsequently states, “We retrain LEMO with all unseen combinations.” (L428-429) This contradicts the definition of unseen combinations and makes Table 3 difficult to interpret as evidence of compositional generalization. If this is a wording error it should be corrected otherwise evaluation on genuinely held-out combinations is needed.

Insufficient validation of temporal continuity at edit boundaries: Section 4.2 claims that the inpainting process “ensures consistency, naturalness” (L193) and Section 4.3 emphasizes “temporal continuity.” (L233-234) However the reported metrics do not directly assess whether edited segments connect smoothly to the surrounding original motion. The human evaluation in Appendix C also focuses on target-emotion preference and lip synchronization rather than transition naturalness. Conditioning on unmasked motion may encourage continuity but does not establish that abrupt changes are avoided at the start and end of an edit.

Minor comment:

  1. Inconsistent description of human lip-sync results: Appendix C.2 states that “EMOTE achieves the highest mean lip-sync score” (L916-917) followed by LEMO with ExpCLIP ranked lowest. However Table 7 reports ExpCLIP at 3.53, LEMO at 3.35 and EMOTE at 2.51 indicating the opposite ordering of ExpCLIP and EMOTE. The authors should reconcile the text with the table and verify the reported method labels.
Questions for authors
  1. How do the authors distinguish emotion-related deviations from errors introduced by TTS, animation generation, and temporal alignment? When the original utterance is already neutral how large is the residual.

  2. Were the unseen upper-region, lip-region action combinations excluded from all data used to retrain LEMO for Table 3? What exactly does “We retrain LEMO with all unseen combinations” (L428-429) mean and how many combinations and samples were assigned to the training and held-out sets?

  3. How does the method maintain smooth transitions between edited and original motion at mask boundaries? Is continuity enforced during sampling or learned solely through reconstruction? Can the authors provide boundary-focused motion analysis?

Reviewer_aTv7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LEMO is a text-guided framework for editing emotions in speech-driven 3D facial animation. It separates motion into upper-face movement, pseudo-neutral lip motion, and expressive lip offsets. An LLM translates editing instructions into structured facial-action descriptors, which guide a diffusion inpainter through region-aware attention adapters. Evaluation includes emotion classification and lip-sync metrics on MEAD and MEAD-OpenVoice, a fine-grained comparison with ExpCLIP, and human ratings.

Critical strengths

The editing task is useful and practically motivated. Allowing users to change specific facial regions and time intervals offers more flexible control than assigning one emotion to an entire sequence. The structured descriptor interface provides an interpretable connection between natural-language requests and motion editing.

The experiments provide promising evidence of editing capability. LEMO achieves competitive emotion recognition and lip-sync scores, and the fine-grained evaluation favors it over ExpCLIP. The paper also examines alignment strategies, conditioning choices, and compositional generalization.

Critical weaknesses

The neutral lip motion is generated using synthesized speech and EMOTE. The difference between this motion and the original may therefore include generation and timing errors, rather than expression alone. Keeping the neutral motion unchanged does not guarantee lip synchronization, because editing the remaining component still changes lip and jaw movements. The paper should compare lip synchronization before and after editing the same animation and test whether separating these components improves preservation. Table 4 also shows mixed results across the two lip-sync metrics, while the ranking described in Appendix C.2 contradicts Table 7.

The attention mask in Section 4.3 blocks unused control tokens, but still allows frames outside the requested interval to attend to active tokens. The authors should explain what prevents unwanted changes elsewhere and measure these changes. The fine-grained comparison includes only ExpCLIP, although the paper describes 3DiFACE as supporting keyframe-based editing. In addition, the VLM judges results against the LLM-generated JSON rather than the original user request, so it may miss errors in interpreting that request. Human validation covers only a small set of LEMO outputs, leaving the reliability of comparisons between methods unclear.

Questions for authors

How much does lip synchronization change before and after editing the same source animation, separately for mouth edits and upper-face edits? Does decomposition improve preservation compared with an otherwise matched editor without decomposition?

What explicitly preserves motion outside the requested region and time interval during inference? Please clarify whether unedited channels are clamped or restored at each sampling step, and report unintended changes outside the edit mask.

Can the fine-grained evaluation include 3DiFACE with matched keyframe or masking controls, and assess outputs against the original natural-language requests? Please also clarify the accuracy of the LLM to JSON translation.

Please resolve the contradiction between the lip-sync ranking described in Appendix C.2 and the scores in Table 7.

What direct evidence distinguishes expression attenuation from incorrect expressions or classifier errors? How do the MEAD, MEAD-OpenVoice comparisons isolate phonetic-content bias from other changes introduced by resynthesis? Please also provide direct evidence for the claimed lip-sync drift when changing emotion and clarify the scope of “content-independent.”

Reviewer_sq3vOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose LEMO, a motion editing framework that accounts for prosody and phonetics and addresses their entanglement by editing pre-generated animations rather than controlling expressions during generation. To enable LLM-based editing, the authors discretize the expression offsets into semantic descriptors. This allows an LLM to edit blendshape states without directly predicting their activation values. The authors further introduce an inpainting model that reconstructs masked regions, conditioned on emotion labels and control signals from a ControlNet branch conditioned on these descriptors.

Critical strengths
  • Using LLMs to edit emotions and facial expressions in generated facial animations makes sense, and the approach could generalize well to unseen text instructions at test time in a zero-shot setting.

  • The problem is worth addressing. Generating expressive facial motion that follows speech is challenging when relying solely on paired speech and facial motion data. Disentangling the information related to lip motion and upper-face expressions is particularly difficult.

Critical weaknesses

Formulation: While an LLM enables emotion control, its edits cannot be accurately aligned with the emotions conveyed by the input speech without access to its timing and fine-grained nuances. For example, if the emotion changes during an utterance, the LLM cannot determine the exact timing of that change. Although LLMs enable high-level semantic editing beyond what purely data-driven methods can provide, this does not ensure that the generated facial animations naturally and expressively follow the emotions conveyed by the input speech.

Architecture: While the use of an LLM and the discretization of expression offsets are well motivated, the architectural combination of an inpainting model and ControlNet is insufficiently justified. Both components are reasonable choices, but the motivation for combining them in this particular way is not convincing. Why could a single model not take both control signals and masks as inputs to generate the final results? Moreover, the paper lacks important training details. For example, is ControlNet trained first and then frozen while the inpainting model is trained, or is the inpainting model trained first? The lack of these details makes it difficult for reviewers to assess the contribution of this work.

Qualitative results: The qualitative results are difficult to compare because the videos are provided separately. I strongly encourage the authors to provide a side-by-side comparison video so that reviewers can clearly assess the strengths of the proposed method.

Questions for authors
  • Could the authors more clearly describe the limitations of LLM-based editing in the revised manuscript?

  • Could the authors clarify the training procedures for the inpainting model and ControlNet, including the training order and which components are frozen at each stage?

  • Could the authors explain the motivation behind this architectural arrangement and provide experimental comparisons with alternative implementations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.