acceptodds
Under review as a conference paper at ICLR 2027

Spatial-Query Multimodal Routing for Audio-Driven 3D Gaussian Talking Heads

Abstract

Audio-driven 3D talking heads require a representation that maps global speech signals to localized facial motion while preserving identity and rendering fidelity. Although spatial-query conditioning has been adopted in dynamic 3D Gaussian talking heads, its interaction with local reconstruction objectives and its consistency across identities remain underexplored. Building on GaussianTalker, we study a dynamic 3D Gaussian talking-head framework based on Gaussian-wise spatial-query conditioning. Multi-scale planar features encode the spatial context of each primitive, while a Spatial-Audio Cross-Attention (SACA) module uses these features as queries to route information from audio, eye state, camera pose, and a learnable null condition. The resulting Gaussian-specific conditioning predicts residual updates to position, scale, rotation, and appearance, enabling speech-driven deformation to vary across facial regions. A progressive training schedule first optimizes global reconstruction and Gaussian structure and then refines local non-rigid deformation with lip-region reconstruction and foreground-mask supervision. Under an aligned-frame evaluation protocol across three identities, the full configuration achieves the highest macro-averaged PSNR and SSIM among the compared configurations, reaching 33.52 dB and 0.936, respectively. In a 36-run 2-by-2 factorial study of SACA and lip supervision, nine matched Full/No-SACA comparisons show that the full model reduces mean LSE-D by 0.411 and increases mean LSE-C by 0.744, with a mean PSNR decrease of 0.181 dB. Across six directed source-to-target identity pairs, ArcFace cosine similarity ranges from 0.910 to 0.947. These results show that the empirical benefit of the complete spatial-query fusion configuration depends on local supervision and identity, improving average audiovisual synchronization under lip supervision while incurring a measurable reconstruction trade-off.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio-driven talking head synthesis, 3D Gaussian splatting, neural rendering, audio-visual generation, facial animation, lip synchronization, multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q7XgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an empirical study of the interaction between complete-module SACA fusion and lip-region reconstruction supervision in a GaussianTalker-based system. A 2×2 factorial experiment covers three identities and three seeds, followed by a single-seed, lip-supervised comparison on fifteen additional identities. The results show a supervision-dependent reversal in the average synchronization ranking of SACA and Mean, heterogeneous effects across identities, and synchronization gains frequently accompanied by reduced mouth reconstruction fidelity. Supplementary records support analysis-level reproduction.

Critical strengths
  1. The core experiment provides a clearly defined comparison. The complete 2×2 design, paired seeds, and identity-level aggregation allow the fusion–supervision interaction to be estimated within the core study. This makes the conditional configuration effects explicit.
  2. The reporting is transparent. Identity-resolved results, negative examples, sensitivity analyses, and archived measurements make the reported variation and limitations relatively easy to inspect.
Critical weaknesses
  1. The fusion comparison provides limited insight into spatial-query routing. In Section 3.5, No-SACA uses LayerNorm applied to the sum of the spatial feature and the averaged condition tokens. It therefore retains spatial conditioning while replacing the complete SACA module, changing condition weighting, feed-forward computation, and active capacity together. The comparison supports a complete-module effect, but does not identify which of these changes accounts for the observed interaction. This limits the explanatory insight provided by the study.
  2. The additional identities do not replicate the central interaction. The complete fusion-by-lip-supervision comparison covers only three identity–video instances. The fifteen-identity extension evaluates Full versus No-SACA with lip supervision enabled, leaving the corresponding lip-off comparisons unavailable. The extension therefore broadens the evidence for heterogeneous lip-on effects, but does not establish whether the fusion–supervision interaction extends beyond the three core instances.
  3. The synchronization–reconstruction trade-off remains insufficiently characterized. Section 4.4 shows that synchronization gains often coincide with lower mouth PSNR under the evaluated loss weights. However, the supplementary weight variations do not compare both fusion branches at matched supervision strengths. The results establish this co-occurrence at the reported settings, but leave unclear whether the relative reconstruction cost persists after adjusting the lip-loss weight. This limits the guidance provided on choosing between the two fusion configurations.
Questions for authors

Please prioritize the controlled fusion comparison in Question 1 and the evaluation clarification in Question 4. Questions 2 and 3 describe targeted extensions that would strengthen the assessment, if feasible within the rebuttal period.

  1. Controlled fusion comparison. Does the interaction persist when adaptive attention is replaced with uniform weights while retaining the value/output projections, FFN, residual connections, and normalization? As an exploratory rebuttal check, this control could be evaluated on the three core identities at seed 1, with lip supervision on and off, requiring six additional runs. Please report the LSE-D/LSE-C conditional effects and interactions relative to the corresponding Full configurations.
  2. Extending the factorial evaluation. If feasible, could the factorial comparison be completed for three additional identities selected using a stated criterion independent of their observed Full–No-SACA gains? Adding the two missing lip-off configurations at seed 1 would require six additional runs. Do the identity-level interactions retain the directions observed in the core study? If this evaluation is unavailable, please explain what evidence supports extending the interpretation of the interaction beyond the three core instances, or explicitly restrict that interpretation to the core study.
  3. Matched lip-supervision weights. The supplement includes SACA results for May/seed-1 with lip-loss weights of 0.2 and 0.8. Could the corresponding Mean configurations be evaluated, allowing both branches to be compared at weights of 0.2, 0.4, and 0.8? Please report synchronization and mouth reconstruction metrics to assess whether the observed trade-off persists across these settings.
  4. Synchronization evaluation protocol. For Kjell, Tables 6 and 14 both report Full/seed-1 results on 238 frames, but give different LSE-D/LSE-C scores: 14.432/1.020 and 14.053/1.095, respectively. What accounts for this difference? Please clarify whether the checkpoint, rendered inputs, audio, face processing, or scorer settings differ, and provide the ground-truth synchronization scores under the same evaluation protocol.
Reviewer_NaTGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how spatial-query fusion interacts with lip reconstruction supervision in GaussianTalker-based talking heads. A factorial study compares the complete SACA module with mean fusion under different supervision settings. Additional identities reveal variable synchronization outcomes and a frequent cost in mouth reconstruction. The contribution is an empirical analysis showing that the relative performance of these configurations depends on supervision and identity.

Critical strengths
  1. The factorial design directly examines whether lip supervision changes the relative performance of the fusion configurations.

  2. The identity-resolved evaluation retains unfavorable outcomes and reports mouth reconstruction alongside synchronization. This provides useful evidence that average synchronization gains can conceal inconsistent results and reconstruction costs.

Critical weaknesses
  1. The architecture and training strategy are inherited from GaussianTalker, and the contribution rests on what the empirical analysis establishes. Replacing SACA with mean fusion also changes active capacity and feed-forward computation. The results do not isolate the role of spatial-query routing, limiting the explanatory advance.

  2. The central interaction is evaluated on a small identity set at a fixed lip-loss weight. The additional identities cover only the lip-supervised comparison and cannot establish whether the interaction generalizes. The average synchronization benefit is sensitive to identity composition.

  3. Although TalkingGaussian and SyncTalk are discussed in the related work, neither is included in the experiments. Several recent Gaussian-based methods are also missing. Without these comparisons, it is difficult to judge how much improvement the proposed method actually brings over existing methods.

Questions for authors
  1. Does the interaction persist when condition weights are fixed while retaining SACA’s learned projections and feed-forward layers?

  2. Can the interaction be checked on additional identities and under a nearby lip-loss weight?

  3. How does the full configuration compare with TalkingGaussian and SyncTalk under the same evaluation protocol? This comparison would help establish the practical significance of the reported gains.

Reviewer_sV2JOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes spatial-audio cross-attention (SACA) and lip reconstruction supervision in GaussianTalker. All experiments use this existing framework, examining its behavior rather than introducing a new talking-head method. The authors compare SACA with mean fusion, with and without lip supervision, across three identities and three training seeds, and extend the lip-supervised comparison to fifteen additional identities. The results suggest that, within this framework, SACA’s synchronization gains depend on supervision and identity and can come at the cost of reconstruction quality.

Critical strengths
  1. The paper examines how SACA and lip supervision work together, rather than testing each component separately. This helps show that the effect of the fusion module depends on the training objective.
  2. The experiments include multiple seeds and per-identity results, showing cases where SACA helps and where it does not.
  3. The paper reports both synchronization and regional reconstruction quality, highlighting that better synchronization scores do not always mean better mouth reconstruction.
Critical weaknesses
  1. The contribution is somehow narrow. The model and training procedure largely follow GaussianTalker, and the paper mainly adds experiments on it. More identities do not address whether the findings apply to other models. Although a single-model study might still be valuable, but here the analysis does not explain why the effects occur or offer much guidance on how to improve the model. The work therefore reads more like an extended ablation of GaussianTalker than a substantial standalone contribution.
  2. The No-SACA comparison changes more than attention. Mean fusion removes the attention mechanism along with other learnable transformations and feed-forward computation. It is unclear whether the differences come from spatial-query routing, model capacity, or optimization. Keeping the rest of the module unchanged and replacing only the attention weights would be a cleaner test.
  3. The main finding is tested on only three identities. The fifteen additional identities are evaluated with lip supervision enabled and only one seed, so they do not verify the interaction between fusion and supervision. The gains also vary considerably across subjects. For example, removing May eliminates the average lip-on LSE-D improvement. Testing the full comparison on more identities and varying the lip-loss weight would make the conclusion more convincing.
  4. The results do not show a clear practical advantage. In Table 5, Full has better PSNR and SSIM than GaussianTalker, but worse LPIPS and synchronization scores. Compared with mean fusion, it also sacrifices reconstruction quality for average synchronization gains. The paper reports these trade-offs, but it remains unclear which configuration produces better-looking videos or when SACA should be preferred.
  5. There are no video comparisons. Static frames cannot show whether the lips follow the audio or whether the motion is smooth. Videos are particularly important here because synchronization is central to the paper’s analysis.
Questions for authors
  1. Do the findings also hold for other talking-head models or fusion modules, or are they specific to this GaussianTalker configuration?
  2. Does the interaction with lip supervision remain when the mean-fusion baseline retains comparable capacity and the same feed-forward layers as SACA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.