- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents AudioMouth, a speech-driven mouth animation method. It frames animation as an "articulation-grounded structured generation" task within a multimodal language model. Specifically, the model takes audio, transcripts, phonemes, and a set of fixed phoneme-to-articulation rules as inputs, and generates JSON trajectories for 32 ARKit blendshape controls. To make this work, the authors combine semantic segmentation with shared temporal boundaries, coefficient reweighting during SFT, and further refine the outputs using GRPO. The GRPO stage uses several reward signals, including coefficient matching, temporal differences, and a Simplified Lip-Geometry Discrepancy (SLMD) reward. The method is evaluated on a filtered English BEAT dataset (with speaker-disjoint splits) using 60 test scripts across 2,946 segments, leveraging dataset-provided linguistic annotations during test time.
Critical strengths
-
The proposed articulation guidance is well-motivated and shows clear, controlled benefits. Using phonemes and named ARKit controls together creates a neat semantic interface, making it easy to incorporate qualitative articulatory priors directly. The ablation study in Figure 4(a) is solid. The "w/o Guidance" version keeps everything else intact—the backbone, training data, transcripts, phonemes, timing instructions, and output constraints—and only drops the articulation-guidance block. Adding this guidance back drops the SFT's FD score from 53.7 to 11.3 and the WInD score from 62.4 to 17.1 (detailed on pages 8-9 and Appendix C.2). This clearly shows that the guidance block does more than just feed extra linguistic data into the generator. Also, these performance gains aren't just rescuing an otherwise useless model. For instance, the "Audio Only" baseline already gets an MSE of ~0.00729 and an LMD of ~6.9, which actually beat the best baseline numbers in Table 1 (0.020 and 7.648 respectively), even though its FD score is quite high at 83.6.
-
The proposed coefficient reweighting and geometric feedback also have good empirical support. In Figure 4(b), we can see that reweighting drops the MSE from around 0.01223 to 0.00598 and improves all five metrics. The final configuration in Table 1 achieves an MSE of 0.006, an FD of 10.820, and an LMD of 6.249. The SLMD metric is well-defined: it calculates joint vertex displacements using a fixed blendshape basis and then takes a non-squared L2 norm. This is theoretically different from the 2D Lip Landmark Distance (LMD) calculated from rendered videos (see pages 5-6, Eqs. 5-6). The authors ablated SLMD in Figure 4(c), and Figure 6 shows that GRPO still brings gains even without smoothing. Regarding the reward implementation details in the supplementary material, using an integer domain of 0 to 10 makes sense, and it explains why the MSE scale of 0.660 there isn't directly comparable to the evaluation errors on 0–1 coefficients.
Critical weaknesses
-
The supplementary videos raise a direct concern about whether AudioMouth achieves its central goal: mouth movements that match the accompanying speech. In the supplied demonstrations, the generated mouth movements appear poorly synchronized with the pronunciation, making the animation unconvincing as speech-driven motion. This matters more than a small difference in average benchmark scores. Table 1 reports strong results, including FD of 10.820 and LMD of 6.249, but FD measures motion-distribution similarity, while LMD measures average distances to reference mouth landmarks—not audiovisual synchrony directly. LMD is temporally paired and therefore does contain relevant information, but its average value does not resolve the apparent mismatch in the demonstrations. The videos provide qualitative evidence, not a measured failure rate, and they do not by themselves establish whether the problem originates in generation, trajectory reconstruction, or audio/video export. Nevertheless, until this discrepancy is explained and synchronization is directly validated, I remain unconvinced that the method reliably produces the speech-aligned articulation it claims.
-
The comparison does not isolate the benefit of the proposed generator from the benefit of richer inputs and dataset adaptation. AudioMouth receives audio, transcripts, and phonemes, whereas all four external baselines receive only audio. Only SAiD is fine-tuned on the same training split; the other three use released weights (Section 4; Appendix C.6). Thus, the large improvements in Table 1 may reflect both the method and differences in information or training conditions. Figure 4 shows that guidance and reweighting help AudioMouth, but it does not answer the simpler question: would a conventional mouth-motion predictor perform similarly if given the same data and linguistic inputs? A targeted matched comparison would make the claimed advantage much easier to assess; I am not asking the authors to prove that an LLM is necessary or to match every existing baseline exhaustively.
Questions for authors
-
Could you explain the apparent speech–mouth mismatch in the supplementary videos and identify whether it comes from the predicted trajectories or from reconstruction/rendering/audio-video export? Please map the demonstrations to the evaluated configurations and provide short, uncut comparisons of the reference, AudioMouth, and a baseline using the same audio, rendering, and playback rate, with any offset correction disclosed. A blinded assessment of lip-sync and articulatory plausibility on a representative held-out subset, alongside a suitable direct synchronization measure, would help establish whether the issue is confined to presentation or affects the generated motion itself. Please relate those results to the reported LMD/FD scores.
-
With the same training split, audio+transcript+phoneme inputs, 32 ARKit controls, and post-processing conditions, how does AudioMouth compare with a conventional regression-based or token-generative mouth-motion model? If such a comparison is unavailable, can a controlled input/adaptation comparison clarify how much of the advantage in Table 1 comes from the generator rather than additional linguistic information and training? Please include synchronization quality, not just aggregate coefficient or distribution errors.
3 comments
Thank you for the response and the additional experiments. The matched-input regression baseline helps address my concern about comparison fairness, and the explanation of the rendering issue clarifies the apparent mismatch in the original videos. However, I am still not fully convinced about the actual lip-sync quality. A small estimated AV offset is useful evidence for temporal alignment, but it does not necessarily show that the mouth shapes match the pronounced sounds or look convincing.
Updated Supplementary Videos and Mouth-Shape Evaluation
Dear Reviewer,
Thank you for your follow-up. We are pleased that the matched-input comparison and rendering clarification have helped address your earlier concerns. We agree that a small AV offset establishes temporal alignment but is insufficient to demonstrate convincing, speech-appropriate mouth shapes.
We have rerun inference and updated the supplementary comparison videos. We reran AudioMouth using the standard inference pipeline and regenerated its rendered outputs. The revised supplementary videos for 9_miranda_0_11_11 and 9_miranda_0_17_17 present continuous comparisons with ground truth and all four existing baselines, accompanied by the corresponding speech.
These updated sequences allow direct inspection of mouth openings, closures, and transitions against the spoken audio throughout each clip. We respectfully invite you to revisit these regenerated comparisons when assessing the actual speech–mouth correspondence and visual quality.
AudioMouth's Rebuttal: Clarification of Video Synchronization and Additional Controlled Comparisons
Dear Reviewer,
Thank you for emphasizing audiovisual synchronization. We agree that the demonstration issue requires direct synchronization evidence in addition to coefficient and distributional metrics. We have identified its cause and provide additional synchronization results and a controlled regression comparison below.
1. What caused the mismatch in the supplementary videos?
The mismatch originated from rendering-engine settings left over from other rendering jobs. We apologize for failing to catch this error in the supplementary material. After correcting these settings, we evaluated the rendered outputs without any additional audio or video time shifts.
2. Does the final model produce synchronized mouth motion?
The final model achieves a mean absolute AV offset below one evaluation frame for each of the ten BEAT IDs evaluated.
The reported mean absolute offsets range from 1.818 to 26.667 ms. These measurements directly assess audiovisual timing, complementing FD and LMD, which assess motion distributions and mouth-landmark accuracy.
3. Does the advantage persist against direct regression with matched inputs?
Yes. AudioMouth achieves lower errors on all five metrics than the matched-input regression baseline, and this advantage is already present before GRPO.
We added Qwen3-Omni Regression to address your concern about whether a conventional motion predictor could achieve comparable results when given the same information and training data. It uses the same backbone architecture, training split, audio/transcript/phoneme inputs, and 32 ARKit controls. It also directly optimizes the motion objective used in our GRPO reward, incorporating coefficient, temporal, and geometric supervision.
The comparison below includes both stages of AudioMouth. All methods are evaluated on the same 60 test scripts, with smoothing disabled throughout.
These results address three aspects of your concern:
This comparison supports AudioMouth’s overall generation and training approach under matched input information. Architecture, adaptation, and optimization differences are documented in Sec. 4 and Appendix C.7, with the smoothing ablation in Appendix D. Direct audiovisual synchronization measurements for the final model are reported separately in Point 2.