acceptodds
Under review as a conference paper at ICLR 2027

AudioMouth: Articulation-Grounded Structured Generation for Speech-Driven Mouth Animation

Abstract

Speech-driven mouth animation aims to produce facial motion synchronized with speech, and existing audio-driven methods generally learn this mapping from acoustic features while leaving the articulatory structure of speech implicit in the waveform. Such implicit learning forces the generator to infer which mouth shape a sound requires from acoustics alone, and token-level objectives do not explicitly supervise the dynamics or geometry of complete motion sequences. In this paper, we aim to advance speech-driven mouth animation from implicit acoustic regression to explicit articulation-grounded generation. To this end, we propose AudioMouth, which formulates mouth animation as schema-constrained sequence generation with a multimodal language model. AudioMouth first segments long recordings into semantically coherent units sharing temporal boundaries across audio, transcript, phonemes, and target motion, so each phoneme sequence is paired with motion from the same segment. Conditioned on these units, AudioMouth generates fixed-schema ARKit coefficient sequences under explicit phoneme-to-articulation grounding, and reweights coefficient-value tokens to reduce the relative contribution of frequent near-zero targets to supervision. To supervise the trajectory as a whole, AudioMouth further refines the generator with Group Relative Policy Optimization, whose reward measures coefficient accuracy, temporal dynamics, and a simplified lip-geometry discrepancy that reads 3D mouth deformation out of a fixed blendshape basis without rendering or landmark detection. On our selected subject-disjoint BEAT split, using the provided linguistic annotations, AudioMouth improves every reported metric over the four evaluated baseline configurations, reducing Fr\'echet distance by 57% and coefficient MSE by 70% relative to the best baseline in each metric. Our ablations support the benefits of articulation guidance for distributional fidelity, balanced supervision for coefficient reconstruction, and geometric feedback for reduced across-script FD dispersion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech-driven facial animation, ARKit blendshapes, multimodal large language models, phoneme-aware structured generation, motion-aware reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qjTYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AudioMouth, a speech-driven mouth animation method. It frames animation as an "articulation-grounded structured generation" task within a multimodal language model. Specifically, the model takes audio, transcripts, phonemes, and a set of fixed phoneme-to-articulation rules as inputs, and generates JSON trajectories for 32 ARKit blendshape controls. To make this work, the authors combine semantic segmentation with shared temporal boundaries, coefficient reweighting during SFT, and further refine the outputs using GRPO. The GRPO stage uses several reward signals, including coefficient matching, temporal differences, and a Simplified Lip-Geometry Discrepancy (SLMD) reward. The method is evaluated on a filtered English BEAT dataset (with speaker-disjoint splits) using 60 test scripts across 2,946 segments, leveraging dataset-provided linguistic annotations during test time.

Critical strengths
  1. The proposed articulation guidance is well-motivated and shows clear, controlled benefits. Using phonemes and named ARKit controls together creates a neat semantic interface, making it easy to incorporate qualitative articulatory priors directly. The ablation study in Figure 4(a) is solid. The "w/o Guidance" version keeps everything else intact—the backbone, training data, transcripts, phonemes, timing instructions, and output constraints—and only drops the articulation-guidance block. Adding this guidance back drops the SFT's FD score from 53.7 to 11.3 and the WInD score from 62.4 to 17.1 (detailed on pages 8-9 and Appendix C.2). This clearly shows that the guidance block does more than just feed extra linguistic data into the generator. Also, these performance gains aren't just rescuing an otherwise useless model. For instance, the "Audio Only" baseline already gets an MSE of ~0.00729 and an LMD of ~6.9, which actually beat the best baseline numbers in Table 1 (0.020 and 7.648 respectively), even though its FD score is quite high at 83.6.

  2. The proposed coefficient reweighting and geometric feedback also have good empirical support. In Figure 4(b), we can see that reweighting drops the MSE from around 0.01223 to 0.00598 and improves all five metrics. The final configuration in Table 1 achieves an MSE of 0.006, an FD of 10.820, and an LMD of 6.249. The SLMD metric is well-defined: it calculates joint vertex displacements using a fixed blendshape basis and then takes a non-squared L2 norm. This is theoretically different from the 2D Lip Landmark Distance (LMD) calculated from rendered videos (see pages 5-6, Eqs. 5-6). The authors ablated SLMD in Figure 4(c), and Figure 6 shows that GRPO still brings gains even without smoothing. Regarding the reward implementation details in the supplementary material, using an integer domain of 0 to 10 makes sense, and it explains why the MSE scale of 0.660 there isn't directly comparable to the evaluation errors on 0–1 coefficients.

Critical weaknesses
  1. The supplementary videos raise a direct concern about whether AudioMouth achieves its central goal: mouth movements that match the accompanying speech. In the supplied demonstrations, the generated mouth movements appear poorly synchronized with the pronunciation, making the animation unconvincing as speech-driven motion. This matters more than a small difference in average benchmark scores. Table 1 reports strong results, including FD of 10.820 and LMD of 6.249, but FD measures motion-distribution similarity, while LMD measures average distances to reference mouth landmarks—not audiovisual synchrony directly. LMD is temporally paired and therefore does contain relevant information, but its average value does not resolve the apparent mismatch in the demonstrations. The videos provide qualitative evidence, not a measured failure rate, and they do not by themselves establish whether the problem originates in generation, trajectory reconstruction, or audio/video export. Nevertheless, until this discrepancy is explained and synchronization is directly validated, I remain unconvinced that the method reliably produces the speech-aligned articulation it claims.

  2. The comparison does not isolate the benefit of the proposed generator from the benefit of richer inputs and dataset adaptation. AudioMouth receives audio, transcripts, and phonemes, whereas all four external baselines receive only audio. Only SAiD is fine-tuned on the same training split; the other three use released weights (Section 4; Appendix C.6). Thus, the large improvements in Table 1 may reflect both the method and differences in information or training conditions. Figure 4 shows that guidance and reweighting help AudioMouth, but it does not answer the simpler question: would a conventional mouth-motion predictor perform similarly if given the same data and linguistic inputs? A targeted matched comparison would make the claimed advantage much easier to assess; I am not asking the authors to prove that an LLM is necessary or to match every existing baseline exhaustively.

Questions for authors
  1. Could you explain the apparent speech–mouth mismatch in the supplementary videos and identify whether it comes from the predicted trajectories or from reconstruction/rendering/audio-video export? Please map the demonstrations to the evaluated configurations and provide short, uncut comparisons of the reference, AudioMouth, and a baseline using the same audio, rendering, and playback rate, with any offset correction disclosed. A blinded assessment of lip-sync and articulatory plausibility on a representative held-out subset, alongside a suitable direct synchronization measure, would help establish whether the issue is confined to presentation or affects the generated motion itself. Please relate those results to the reported LMD/FD scores.

  2. With the same training split, audio+transcript+phoneme inputs, 32 ARKit controls, and post-processing conditions, how does AudioMouth compare with a conventional regression-based or token-generative mouth-motion model? If such a comparison is unavailable, can a controlled input/adaptation comparison clarify how much of the advantage in Table 1 comes from the generator rather than additional linguistic information and training? Please include synchronization quality, not just aggregate coefficient or distribution errors.

3 comments
AuthorsOpenReview ↗

AudioMouth's Rebuttal: Clarification of Video Synchronization and Additional Controlled Comparisons

Dear Reviewer,

Thank you for emphasizing audiovisual synchronization. We agree that the demonstration issue requires direct synchronization evidence in addition to coefficient and distributional metrics. We have identified its cause and provide additional synchronization results and a controlled regression comparison below.

1. What caused the mismatch in the supplementary videos?

The mismatch originated from rendering-engine settings left over from other rendering jobs. We apologize for failing to catch this error in the supplementary material. After correcting these settings, we evaluated the rendered outputs without any additional audio or video time shifts.

2. Does the final model produce synchronized mouth motion?

The final model achieves a mean absolute AV offset below one evaluation frame for each of the ten BEAT IDs evaluated.

BEAT IDMean absolute AV offset (frames)Offset (ms)Confidence
9_miranda0.1774537.0985.5003
5_stewart0.25802310.3215.3367
12_zhao0.51428620.5715.5522
13_lu0.30769212.3085.0115
14_zhang0.1500006.0005.6962
16_jorge0.0454551.8185.2037
17_itoi0.0909093.6365.5087
27_yingqing0.63636425.4554.5883
28_tiffnay0.66666726.6675.2351
30_katya0.1538466.1544.6368

The reported mean absolute offsets range from 1.818 to 26.667 ms. These measurements directly assess audiovisual timing, complementing FD and LMD, which assess motion distributions and mouth-landmark accuracy.

3. Does the advantage persist against direct regression with matched inputs?

Yes. AudioMouth achieves lower errors on all five metrics than the matched-input regression baseline, and this advantage is already present before GRPO.

We added Qwen3-Omni Regression to address your concern about whether a conventional motion predictor could achieve comparable results when given the same information and training data. It uses the same backbone architecture, training split, audio/transcript/phoneme inputs, and 32 ARKit controls. It also directly optimizes the motion objective used in our GRPO reward, incorporating coefficient, temporal, and geometric supervision.

The comparison below includes both stages of AudioMouth. All methods are evaluated on the same 60 test scripts, with smoothing disabled throughout.

Method, without smoothingMSE ↓MAE ↓FD ↓WInD ↓LMD ↓
Qwen3-Omni Regression0.0089130.06110915.10219.2906.400
AudioMouth — Stage 2 SFT0.0060650.05380211.51017.3356.307
AudioMouth — SFT + GRPO0.0057890.05319711.05516.8306.252

These results address three aspects of your concern:

  • The advantage persists with matched linguistic information and training data. Even before GRPO, AudioMouth’s SFT model improves all five metrics over direct regression. The difference therefore remains when the regression baseline also receives transcripts, phonemes, and task-specific motion supervision.
  • GRPO further improves the generated motion. Refinement improves all five SFT metrics. Relative to direct regression, the final model reduces MSE by 35.0%, FD by 26.8%, and WInD by 12.8%, alongside a smaller 2.3% improvement in LMD. The strongest improvements concern coefficient accuracy and motion distributions, while the landmark improvement is more modest.
  • These gains persist without smoothing. Disabling smoothing for every method shows that the observed improvements are present in the model outputs themselves.

This comparison supports AudioMouth’s overall generation and training approach under matched input information. Architecture, adaptation, and optimization differences are documented in Sec. 4 and Appendix C.7, with the smoothing ablation in Appendix D. Direct audiovisual synchronization measurements for the final model are reported separately in Point 2.

Thank you for the response and the additional experiments. The matched-input regression baseline helps address my concern about comparison fairness, and the explanation of the rendering issue clarifies the apparent mismatch in the original videos. However, I am still not fully convinced about the actual lip-sync quality. A small estimated AV offset is useful evidence for temporal alignment, but it does not necessarily show that the mouth shapes match the pronounced sounds or look convincing.

AuthorsOpenReview ↗

Updated Supplementary Videos and Mouth-Shape Evaluation

Dear Reviewer,

Thank you for your follow-up. We are pleased that the matched-input comparison and rendering clarification have helped address your earlier concerns. We agree that a small AV offset establishes temporal alignment but is insufficient to demonstrate convincing, speech-appropriate mouth shapes.

We have rerun inference and updated the supplementary comparison videos. We reran AudioMouth using the standard inference pipeline and regenerated its rendered outputs. The revised supplementary videos for 9_miranda_0_11_11 and 9_miranda_0_17_17 present continuous comparisons with ground truth and all four existing baselines, accompanied by the corresponding speech.

These updated sequences allow direct inspection of mouth openings, closures, and transitions against the spoken audio throughout each clip. We respectfully invite you to revisit these regenerated comparisons when assessing the actual speech–mouth correspondence and visual quality.

Reviewer_j1vMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AudioMouth formulates speech-driven mouth animation as autoregressive generation of 32 ARKit coefficient trajectories using Qwen3-Omni. The model receives audio, transcripts, phonemes, and fixed natural-language instructions connecting phoneme categories to named facial controls, and generates quantized coefficients in a prescribed JSON schema. Training consists of coefficient-value-weighted supervised fine-tuning followed by GRPO with rewards measuring coefficient accuracy, temporal dynamics, and mouth geometry through a fixed blendshape basis. On a selected speaker-disjoint BEAT split, using provided linguistic annotations at test time, the method outperforms four evaluated system configurations on five metrics. Ablations investigate articulation guidance, coefficient reweighting, geometric feedback, and inference-time smoothing.

Critical strengths
  • The articulation-guidance contribution is supported by a controlled ablation. Section 5.1 holds the backbone, training data, objective, and post-processing fixed while varying conditioning. Adding phoneme-to-control guidance to audio, transcripts, and phonemes reduces FD from approximately 53.7 to 11.3. This is consequential evidence: the improvement cannot be attributed solely to a larger backbone or access to additional linguistic inputs. The shared vocabulary between instructions and named ARKit controls also makes the design interpretable.

  • The geometric reward has a sound motivation and targeted empirical support. SLMD accounts for how multiple coefficients jointly deform the mouth surface, which coefficient-wise errors do not directly capture, while avoiding rendering within the sampling loop. The ablation removing SLMD supports its usefulness within the proposed refinement procedure. The additional comparison with and without smoothing helps distinguish learned refinement from inference-time filtering.

Critical weaknesses
  • The evaluation does not sufficiently establish the proposed method’s comparative advantage under matched conditions or automatically obtained inputs. All four baselines receive audio alone, whereas AudioMouth receives provided transcripts and phonemes at test time; three baselines are also evaluated without adaptation to the selected training split. The authors explicitly acknowledge these differences, and Table 1 remains a valid comparison of the stated configurations. Nevertheless, it does not establish how much advantage the proposed formulation provides over a strong motion generator with comparable linguistic information and training access. The internal conditioning ablation addresses a different question and does not resolve this comparison. Furthermore, linguistic annotations determine both conditioning and segment construction, yet the method is not evaluated with automatically estimated transcripts, phonemes, and timing information. These gaps substantially limit the evidence for the approach’s usefulness beyond the annotation-assisted benchmark.

  • The additional benefit and reproducibility of motion-aware refinement are insufficiently established. Section 5.3 reports approximately 4% improvements in mean MSE and FD over SFT. Such improvements can be meaningful, but repeated training runs or uncertainty estimates are needed to assess their reliability, particularly given the substantial variation in some ablated configurations. The reported standard deviations describe variation across test scripts, not variation across training runs. Important reward details are also underspecified: the exact temporal discrepancy norms and reductions, coefficient units used inside the reward, and the procedure used to obtain the reference-median normalization scales. These details affect candidate rankings after nonlinear score conversion and are necessary for reproduction. Finally, the current comparisons do not establish the benefit of GRPO relative to a training-budget-matched continuation of SFT or direct motion supervision, leaving the value of the additional optimization stage uncertain.

Questions for authors
  • How does performance change when transcripts, phonemes, and segmentation timestamps are obtained automatically from the test audio, without access to the provided linguistic annotations? Does the benefit over the audio-only variant persist under this setting?

  • What evidence is available against a motion generator trained on the same split with comparable transcript and phoneme conditioning? A comparison using direct coefficient prediction and motion supervision would help determine the additional value of autoregressive JSON generation and GRPO, beyond the value of linguistic conditioning.

  • Please specify the exact definitions and reductions of the velocity and acceleration discrepancies, the coefficient units used for each reward term, and which prediction–reference pairs determine the normalization scales. How reproducible are the SFT-to-GRPO and SLMD-ablation results across independent refinement runs?

  • What are the test-time schema-validity and exact-frame-count rates, and how are invalid or wrong-length outputs handled when computing the reported metrics? Are all 60 test scripts retained? Appendix C.4 evaluates only the shared prefix for length errors up to 20%; consequently, omitting a difficult suffix could improve the motion reward by more than the length penalty. Was this possibility observed or explicitly checked?

3 comments
AuthorsOpenReview ↗

AudioMouth‘s Rebuttal: Automatic Inference, Reproducibility, and GRPO Controls

Dear Reviewer j1vM,

Thank you for identifying the comparison and reproducibility gaps. We added targeted experiments and clarified the evaluation protocol. Major additions are highlighted in blue.

1. Automatic inputs: does the advantage over Audio Only persist?

Yes. The complete automatic pipeline improves all five metrics over both audio-only inference and the separately trained Audio Only SFT variant.

Input / modelMSE ↓MAE ↓FD ↓WInD ↓LMD ↓
Audio only / final model0.0080450.05454417.84525.4327.824
Automatic / final model0.0058410.0484949.94415.5676.823
Annotations / final model0.0057130.05287810.82016.5136.249
Separately trained Audio Only SFT0.0072880.05989683.64093.2306.938

The first three rows use the same final-model weights on all 60 English test scripts. The automatic setting obtains transcripts, phonemes, and segmentation from raw audio. Its FD and WInD are 44.3% and 38.8% lower than audio-only inference. Full statistics and input construction are in Appendix E.

2. Direct prediction: does the advantage remain with matched conditioning?

Yes. AudioMouth outperforms the added Qwen3-Omni Regression baseline on all five metrics without smoothing. Both use the same backbone architecture, data split, and audio-linguistic inputs; the baseline directly predicts continuous coefficients with motion supervision.

Method, without smoothingMSE ↓MAE ↓FD ↓WInD ↓LMD ↓
Qwen3-Omni Regression0.0089130.06110915.10219.2906.400
AudioMouth0.0057890.05319711.05516.8306.252

This supports the complete generation/training approach under matched input information. Architecture, adaptation, and objective differences are detailed in Sec. 4 and Appendix C.7; the smoothing ablation is in Appendix D.

3. Reward specification and normalization

The revised appendix specifies the reward definitions and their validation-based calibration. Exact temporal losses, reductions, units, geometric computation, and normalization procedures are collected in Appendix C.4, including the prediction–reference pairs used to obtain the fixed scales.

4. Reproducibility and the benefit over continued SFT

Every full-GRPO run improves all five means over Stage 2 SFT and its corresponding w/o-SLMD run across three independent training runs.

Training conditionMSE ↓MAE ↓FD ↓WInD ↓LMD ↓
Stage 2 SFT0.0059780.05346711.28217.0526.297
Continued SFT0.0060450.04738714.59823.0916.785
GRPO w/o SLMD, three-run mean0.0057340.05292314.35420.2456.342
Full GRPO, three-run mean0.0054120.0484359.92915.3236.212

Across training runs, FD is 9.929 ± 0.788 for full GRPO versus 14.354 ± 0.075 without SLMD; these deviations measure variation across training runs, distinct from the original across-script deviations.

Continued SFT improves MAE but worsens the other four metrics. It therefore does not reproduce the observed distributional and geometric gains. Per-run results, uncertainty statistics, and control details are in Appendix D.1.

5. Output validity and evaluation coverage

All 60 test scripts are retained, and metrics use full-duration reconstructed scripts. In one complete inference run:

  • Valid JSON: 2,969/2,970 outputs (99.9663%).
  • Exact frame count: 2,968/2,970 outputs (99.9327%).

We also corrected the reported unit count from 2,946 to 2,970; the 60-script split is unchanged. Appendices C.4–C.5 specify validity checks and handling of failed or wrong-length outputs. Shared-prefix scoring is confined to training and does not crop the test trajectories.

Thank you for the detailed response and additional experiments. The automatic-input evaluation, clarified reward definitions, and repeated refinement runs substantially address several of my concerns. The regression comparison is also helpful, although the difference in backbone adaptation limits how confidently the gains can be attributed to the proposed generation formulation. The training-budget matching for continued SFT also remains insufficiently specified. While the revisions strengthen the empirical evidence, my concern about the comparative advantage under sufficiently controlled training conditions is only partially resolved. Overall, I will retain my rating.

AuthorsOpenReview ↗

Clarification of Backbone Adaptation and Training Budget

Dear Reviewer,

Thank you for your follow-up. We are pleased that the automatic-input evaluation, clarified reward definitions, and repeated refinement runs have substantially addressed several of your concerns. We provide further context for the regression baseline and clarify the training-budget comparison below.

1. Frozen-backbone regression is a literature-supported design with demonstrated capability on our task. We use the pretrained backbone to extract audio-language representations and train a dedicated continuous motion decoder with coefficient, temporal, and geometric supervision.

This design is consistent with Time-LLM (ICLR 2024) and TimeCMA (AAAI 2025), which use frozen language backbones with trainable continuous forecasting components. Tang et al. (TMLR 2025) also investigate regression using fixed LLM representations and downstream predictors. These precedents motivate preserving pretrained representations while learning a task-specific regression mapping.

Importantly, our regression baseline itself outperforms all four existing baseline configurations on all five metrics, demonstrating useful motion-prediction capability on this benchmark. With smoothing disabled, AudioMouth’s SFT stage already improves FD from 15.102 to 11.510 and WInD from 19.290 to 17.335; GRPO further improves them to 11.055 and 16.830. All five metrics improve at both stages relative to regression.

The comparison therefore provides evidence for the complete generation/training approach against a capable frozen-feature regression baseline. It includes the respective adaptation strategies; isolating the generation formulation alone would additionally require matching backbone adaptation. Architecture and training details are in Sec. 4 and Appendix C.7.

2. Continued SFT was matched to GRPO by approximate training time on the same hardware. Under the reported batch settings, GRPO reached its best checkpoint at approximately 600 training steps, requiring roughly three days on four GPUs. Starting from the same Stage 2 checkpoint, continued SFT received the same approximately three-day, four-GPU training budget.

The results with the same post-processing are:

Training conditionMSE ↓MAE ↓FD ↓WInD ↓LMD ↓
Stage 2 SFT0.0059780.05346711.28217.0526.297
Continued SFT0.0060450.04738714.59823.0916.785
GRPO, reported run0.0057130.05287810.82016.5136.249

Continued SFT improves MAE but worsens the other four metrics relative to Stage 2 SFT. GRPO improves all five over Stage 2 SFT and achieves lower MSE, FD, WInD, and LMD than continued SFT. Under this approximate hardware-time budget, additional supervised training does not reproduce the distributional and geometric improvements obtained through refinement. The three independent refinement runs further support the repeatability of these gains (Appendix D.1).

Reviewer_HhdtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AudioMouth, which aims to improve speech-driven mouth animation by explicitly incorporating phoneme information and articulatory priors. The authors divide audio, transcripts, phonemes, and target motion into temporally aligned semantic units, and use a multimodal language model to generate structured ARKit mouth-coefficient sequences. They further apply coefficient-value reweighting to alleviate the supervision bias caused by the large number of near-zero targets. In addition, the authors use GRPO, together with coefficient errors, velocity and acceleration errors, and blendshape-based geometric errors, to further optimize the generated motion trajectories. Experimental results show that the proposed method outperforms the evaluated baselines on the BEAT dataset, suggesting that explicit linguistic and articulatory information, together with motion-level feedback, can be beneficial.

I believe that, in the current era of rapid advances in multimodal large language models, leveraging MLLMs to achieve breakthroughs in 3D talking-head generation is indeed a highly worthwhile direction to explore.

Critical strengths
  1. The paper focuses on explicitly modeling the correspondence between speech, phonemes, and mouth motion. It attempts to alleviate the over-smoothed mouth shapes and insufficient articulation details commonly observed in traditional audio-driven methods, which makes the problem meaningful to study.
  2. The paper uses a multimodal large language model to generate structured ARKit mouth coefficients and further optimizes them with GRPO and geometric feedback. This provides a worthwhile direction for exploring the application of MLLMs to 3D talking-head generation, and the experimental results also demonstrate certain performance advantages.
Critical weaknesses

AudioMouth uses additional inputs, including transcripts and phonemes, and relies on the relatively large Qwen3-Omni model, while some baselines were not retrained under the same input and training conditions. Therefore, the reported performance improvements may not be entirely attributable to the proposed method. There may be concerns regarding the fairness of the comparisons and the sufficiency of the effectiveness validation. I recommend adding comparisons under unified input conditions, experiments using automatically generated ASR transcripts and phoneme recognition results, and more rigorous ablation studies to evaluate the necessity of the articulation grounding and GRPO components. Overall, the authors should further enrich the ablation experiments to better substantiate the contributions of each component.

Questions for authors

as my comment In weakness

3 comments
AuthorsOpenReview ↗

AudioMouth's Rebuttal: Fair Comparisons and Fully Automatic Inference

Dear Reviewer Hhdt,

Thank you for your constructive feedback. We added experiments addressing comparison fairness, automatic inputs, and the contributions of individual components. Major additions are highlighted in blue.

1. Fairness: does the advantage persist with matched inputs and backbone?

Yes. AudioMouth outperforms the new Qwen3-Omni Regression baseline on all five metrics, including when both methods use no smoothing. The comparison uses the same backbone architecture, data split, and audio-linguistic inputs.

Method, without smoothingMSE ↓MAE ↓FD ↓WInD ↓LMD ↓
Qwen3-Omni Regression0.0089130.06110915.10219.2906.400
AudioMouth0.0057890.05319711.05516.8306.252

FD and WInD decrease by 26.8% and 12.8%, respectively. This supports an advantage beyond access to additional linguistic information or inference-time smoothing. The comparison evaluates the complete generation and training approaches. Their implementation differences are documented in Sec. 4 and Appendix C.7, with the smoothing ablation in Appendix D.

2. Automatic inputs: does the benefit survive without dataset annotations?

Yes. Fully automatic inference outperforms all four existing baseline configurations on all five metrics. We evaluated the complete automatic transcript–segmentation–phoneme pipeline on the same 60 English test scripts.

Inputs to the same final modelMSE ↓MAE ↓FD ↓WInD ↓LMD ↓
Audio only0.0080450.05454417.84525.4327.824
Automatic pipeline0.0058410.0484949.94415.5676.823
Dataset annotations0.0057130.05287810.82016.5136.249

Automatic inputs reduce FD by 44.3% and WInD by 38.8% relative to audio-only inference. Compared with dataset annotations, automatic inputs achieve lower MAE, FD, and WInD, while annotations yield lower MSE and LMD. The advantage over the existing baselines therefore persists without annotation-assisted inference. Full results and the automatic pipeline are in Appendix E.

3. Component contributions: what do the ablations establish?

Guidance and reweighting provide distinct gains, and three independent runs reproduce the benefits of GRPO and SLMD.

Training conditionMSE ↓MAE ↓FD ↓WInD ↓LMD ↓
SFT w/o guidance0.0074840.05788553.74362.3566.917
SFT w/o reweighting0.0122260.07472721.66329.2539.011
Full Stage 2 SFT0.0059780.05346711.28217.0526.297
GRPO w/o SLMD0.005734 ± 0.0002360.052923 ± 0.00058814.354 ± 0.07520.245 ± 0.0306.342 ± 0.029
Full GRPO0.005412 ± 0.0002630.048435 ± 0.0038489.929 ± 0.78815.323 ± 1.0376.212 ± 0.033

The first three rows report the SFT ablation means. The last two report mean ± sample SD across three independent training runs.

  • Guidance adds value beyond linguistic inputs: retaining transcripts and phonemes while adding guidance reduces FD by 79.0%.
  • Reweighting benefits both coefficient accuracy and motion quality: all five metrics improve over SFT without reweighting.
  • GRPO and SLMD yield repeatable improvements: every full-GRPO run improves all five means over Stage 2 SFT and its corresponding w/o-SLMD run.

Full ablation settings and per-run results are provided in Secs. 5.1–5.3 and Appendix D.1.

4. Ethics and data use

We added an Ethics Statement clarifying the sources and use of the data and character assets. BEAT was obtained through official registration for non-commercial academic research; supplementary audio comes from BEAT, and character assets come from the official MetaHuman process. We collected no new recordings or facial captures and recruited no evaluation participants. The statement also addresses privacy, authorization, and potential misuse.

I sincerely appreciate the authors’ patience in reading through my comments and their thoughtful responses to my concerns. Their careful explanations also provided helpful information that I had overlooked during my reading of the paper. The authors position articulation grounding as a core mechanism of AudioMouth, claiming that an explicit mapping between phoneme categories and ARKit control parameters improves mouth-motion generation. I acknowledge that the proposed approach achieves overall performance gains, as reflected in improvements across FD, WInD, MSE, MAE, and LMD.

Nevertheless, I remain concerned that the experimental results are insufficient to directly verify whether the model has genuinely learned and executed the claimed phoneme-to-mouth-shape control relationships. Improvements in these metrics may not necessarily result entirely from the model learning such relationships. They could also be attributed, at least in part, to other factors, such as the additional information provided by transcripts and phonemes, improved local modeling enabled by semantic segmentation, and the extra task guidance introduced by the articulation-guidance prompt.

This is my main concern. For talking-head tasks, I hope that approaches based on large-scale models can offer more than changes arising primarily from model scale. Ideally, they should introduce substantive methodological innovations that work in conjunction with the capabilities of large-scale models, rather than relying mainly on increased model parameters to obtain performance improvements. This is precisely why I attach particular importance to whether the experiments in the paper can directly substantiate the mechanism claimed by the authors.

AuthorsOpenReview ↗

Clarification of Articulation Grounding and Controlled Evidence

Dear Reviewer,

Thank you for your follow-up. We are pleased that our additional experiments and clarifications have helped alleviate your earlier concerns, and we appreciate your acknowledgement of the performance gains. We clarify the remaining concern about articulation grounding below.

1. Grounding supplies a qualitative prior connecting phonemes to named controls. As defined in Sec. 3.3, the guidance connects phonetic categories to ARKit controls, such as bilabials to MouthClose and open vowels to JawOpen. The output schema uses these same control names. Audio supplies timing and acoustic cues, while detailed trajectories are learned from training targets. Appendix C.2 explicitly defines the guidance as qualitative heuristics, with no fixed coefficient values or hard constraints on decoded motion.

2. The conditioning ablation tests its contribution beyond linguistic inputs and backbone capacity. In Sec. 5.1, full SFT and w/o guidance share the backbone, training data, segmentation, transcripts, phonemes, objective, and post-processing. The latter also retains general timing, motion, and format instructions; only the articulation-specific block is removed.

SFT conditionMSE ↓MAE ↓FD ↓WInD ↓LMD ↓
Audio Only0.0072880.05989683.64093.2306.938
w/o guidance0.0074840.05788553.74362.3566.917
Full Stage 2 SFT0.0059780.05346711.28217.0526.297

Adding guidance improves all five metrics and reduces FD by 79.0% relative to w/o guidance. Thus, the experiments distinguish the benefit of linguistic inputs from the further benefit of articulation guidance. Additional transcripts, phonemes, segmentation, and model size are held constant in the latter comparison and cannot alone explain its improvement. Retaining general motion instructions also controls for their presence.

3. The additional controls support complementary methodological contributions.

  • Matched backbone and inputs: With smoothing disabled for both methods, AudioMouth improves all five metrics over Qwen3-Omni Regression, reducing FD from 15.102 to 11.055 and WInD from 19.290 to 16.830. This evaluates the complete generation/training approaches; implementation details are in Sec. 4 and Appendix C.7.
  • Automatic inputs: The complete automatic pipeline obtains FD 9.944, WInD 15.567, and LMD 6.823, outperforming all four existing baseline configurations on all five metrics without dataset-provided linguistic annotations (Appendix E).
  • Separate training components: Reweighting reduces SFT MSE from 0.012226 to 0.005978. Across three independent refinement runs, including SLMD reduces mean FD from 14.354 to 9.929 and LMD from 6.342 to 6.212. Every full-GRPO run improves all five metrics over Stage 2 SFT and its corresponding w/o-SLMD run (Secs. 5.2–5.3 and Appendix D.1).

Our claim concerns the added value of an explicit qualitative articulation prior, alongside the separately evaluated supervision and refinement components. The conditioning ablation establishes the benefit of the complete guidance block; isolating each individual phoneme-to-control correspondence remains a finer-grained question. We therefore base the grounding contribution on the controlled conditioning comparison, rather than on overall benchmark gains alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.