- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes LEMO, a motion-level emotion editing framework for speech-driven 3D facial animation motivated by two failure modes: Expression Attenuation and Emotion Mismatch. LEMO decomposes facial motion into upper-area motion, phone-driven neutral lip motion and expressive lip offsets. Through spatial and temporal quantization of blendshape activations it constructs structured region-state descriptors that enable an LLM to interpret free-form text instructions and generate region- and time-specific edits. A diffusion-based temporal inpainter coupled with a region-aware ControlNet applies these edits while keeping neutral lip motion unmasked to preserve lip synchronization. Experiments on MEAD and MEAD-OpenVoice evaluate high-level emotion editing while VLM-based evaluation assesses fine-grained instruction alignment. The authors report improvements in emotion editing and competitive lip synchronization compared with existing methods.
Critical strengths
Relevant formulation of fine-grained expression editing: The paper addresses a practically useful problem: enabling “region-specific or temporally localized edits” in speech-driven facial animation (Section 1). Its motion-level editing formulation allows users to specify facial actions beyond global emotion labels. The “please blink twice” example in Fig. 3 illustrates the intended control over a particular facial region and action count which is valuable for interactive animation editing.
Explicit separation of editable and preserved motion components: The motion inpainting formulation distinguishes upper-area motion and expressive lip offsets from phone-driven neutral lip motion (Sections 4.1–4.2). In particular neutral lip motion is “always kept unmasked” (L207) providing a speech-related motion reference while the model edits expressive components. This is a useful design choice for conditioning expression editing on the underlying articulation.
Promising quantitative results for emotion editing and lip synchronization: Table 1 reports the highest F-mAP and lowest LSE-D among the compared methods on both MEAD and MEAD-OpenVoice. On MEAD, LEMO achieves an F-mAP of 0.398 compared with 0.364 for the strongest baseline on this metric and an LSE-D of 9.83 compared with the best baseline value of 10.90. Table 2 also reports a higher VLM-based instruction alignment score than ExpCLIP (4.02 vs. 3.78). These results provide empirical support for the framework’s editing capabilities while maintaining competitive lip synchronization.
Critical weaknesses
Insufficient validation of the pseudo-neutral decomposition: Section 3.2 describes pseudo-neutral speech as preserving “phone content while removing expressive prosodic cues” (L149) and interprets the residual as “emotion-related deviations.” (L153) However the ASR–TTS–animation pipeline can introduce pronunciation differences and motion generation errors. Motion-Aligned processing aligns phone timing but does not necessarily remove these differences Appendix A.2 also acknowledges “potential interpolation artifacts.” (L672-673) Consequently the residual may contain non-emotional discrepancies in addition to expressive motion. The authors should quantify residual errors on neutral utterances and assess sensitivity to the TTS, animation generator and alignment procedure.
Limited validation of the VLM-based comparative evaluation: Section 5.1 states that Qwen3.7-Plus is used “as the LLM and VLM” (L323) but no evaluation with an alternative VLM is reported. Appendix D.2 provides a useful human-agreement check, yet it covers only 20 edited samples from “our method results only” (L982) with a Spearman correlation of 0.614 (Table 9). This does not establish whether the VLM’s ranking of LEMO against ExpCLIP agrees with human judgment.
Ambiguous training protocol for compositional generalization: Section 5.2 defines an unseen combination as one that “never appears in any training descriptor string” (L428) but subsequently states, “We retrain LEMO with all unseen combinations.” (L428-429) This contradicts the definition of unseen combinations and makes Table 3 difficult to interpret as evidence of compositional generalization. If this is a wording error it should be corrected otherwise evaluation on genuinely held-out combinations is needed.
Insufficient validation of temporal continuity at edit boundaries: Section 4.2 claims that the inpainting process “ensures consistency, naturalness” (L193) and Section 4.3 emphasizes “temporal continuity.” (L233-234) However the reported metrics do not directly assess whether edited segments connect smoothly to the surrounding original motion. The human evaluation in Appendix C also focuses on target-emotion preference and lip synchronization rather than transition naturalness. Conditioning on unmasked motion may encourage continuity but does not establish that abrupt changes are avoided at the start and end of an edit.
Minor comment:
- Inconsistent description of human lip-sync results: Appendix C.2 states that “EMOTE achieves the highest mean lip-sync score” (L916-917) followed by LEMO with ExpCLIP ranked lowest. However Table 7 reports ExpCLIP at 3.53, LEMO at 3.35 and EMOTE at 2.51 indicating the opposite ordering of ExpCLIP and EMOTE. The authors should reconcile the text with the table and verify the reported method labels.
Questions for authors
-
How do the authors distinguish emotion-related deviations from errors introduced by TTS, animation generation, and temporal alignment? When the original utterance is already neutral how large is the residual.
-
Were the unseen upper-region, lip-region action combinations excluded from all data used to retrain LEMO for Table 3? What exactly does “We retrain LEMO with all unseen combinations” (L428-429) mean and how many combinations and samples were assigned to the training and held-out sets?
-
How does the method maintain smooth transitions between edited and original motion at mask boundaries? Is continuity enforced during sampling or learned solely through reconstruction? Can the authors provide boundary-focused motion analysis?