- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents TAVR4D for 4D head reconstruction under visual degradation. The method uses separate bidirectional temporal branches for global head motion and facial articulation, with synchronized audio providing additional information for expression and jaw prediction. A learned visual-reliability gate controls audio-visual fusion and selects a frame for static identity estimation with MICA.
Critical strengths
- The paper presents a well-motivated problem and effectively decomposes the task. It tackles head reconstruction when individual frames are unreliable, sensibly separating global head motion from speech-related articulation. This approach assigns clear, complementary roles to temporal visual context and synchronized audio.
- The evaluation considers head rotation and mouth-motion tracking, not just surface error. Results show significant improvements despite severe visual degradation. Removing audio greatly reduces mouth-motion correlation, highlighting its role in recovering hidden articulation.
Critical weaknesses
- Limited methodological novelty. Relative to AVFace as discussed in the manuscript, the main changes are longer temporal context, separate motion branches, and reliability-gated fusion. These appear to be incremental architectural modifications rather than a substantial methodological advance. The paper does not establish why its specific reliability supervision offers an advantage over conventional learned fusion.
- Most baselines operate on individual visual frames, whereas TAVR4D uses synchronized audio and 100-frame bidirectional context. These comparisons demonstrate a system-level advantage but do not distinguish the proposed design from the benefit of additional information. An audio-visual temporal baseline with matching encoders, training data, augmentation, and context length is needed.
- The gate has modest gains and inconsistent ablation results. Under severe lower-face occlusion, correlation is 0.56 for the full model, 0.54 without gating, and 0.21 without audio, suggesting that audio provides most of the benefit. Gating helps more clearly with mismatched audio, but its clean-input results are inconsistent. Table 5 reports 0.86 without gating, whereas Table 9 reports 0.76, with the full model at 0.84 in both. This discrepancy reverses the conclusion about the gate’s benefit and must be resolved.
- Appendix 7.1 lists loss groups and weights without defining several individual terms, including the mouth-closure, geometry, and motion-range losses. The expression branch is trained before the gate, but the gating values used during this stage are not specified.
Questions for authors
- What is the main technical advance over prior audio-visual reconstruction beyond longer context, separate prediction branches, and scalar gating?
- Why does the no-gate variant obtain a clean-input correlation of 0.86 in Table 5 but 0.76 in Table 9? Do these results use different checkpoints or evaluation settings?
- What gating values are used during training of the expression branch, considering the gate is trained later? Is the reconstruction network updated afterward to incorporate the learned gate? Please provide precise definitions for mouth-closure, geometry, motion-range, and motion-variation losses.
2 comments
Thank you for the reply and additional experiments. The correction of the gate ablation and the explanation of the training procedure address my corresponding concerns, and the interpolation results provide useful additional evidence. However, my main concern about the methodological contribution remains unresolved. The added comparisons do not provide an audio-visual temporal baseline with matched inputs and training conditions. Moreover, the new gate experiment compares two supervised training strategies and reports agreement with the reliability target, without establishing downstream reconstruction benefits over conventional learned fusion. These results therefore do not sufficiently isolate the benefits of the proposed design from those of additional audio and temporal information. I lean to keep my original rating.
Official Comment by Authors
Q1: While our visual and audio encoders are pretrained, our contribution lies in how their information is used for different motion components. Long-range visual context supports global head motion recovery, while synchronized speech provides complementary information for facial articulation. Rather than treating both as a single reconstruction task, TAVR4D uses separate temporal branches and a supervised visual reliability gate for adaptive audio-visual fusion and identity-frame selection. Our ablations support this design: restricting visual context increases rotation error from approximately 6° to 12.6° (Table 11), while removing audio reduces mouth-motion correlation from 0.56 to 0.21 under severe occlusion. The gate provides a smaller improvement under occlusion (0.56 vs. 0.54), but a clearer benefit when visual input is reliable and audio is mismatched (0.82 vs. 0.61).
Q2 – Gate ablation discrepancy. We reran the evaluation using the same checkpoints and evaluation setup and confirmed that the 0.86 reported for w/o gate in Table 5 is a transcription error. The correct value is 0.76, consistent with Table 9; 0.86 belongs to w/o audio. We will correct Table 5 accordingly.
Q3: Gate training and loss definitions. Expression reconstruction initially uses fixed audio-visual fusion (gt=0.5). We then train the reliability gate separately using DINOv2 clean/corrupted feature similarity, with reconstruction frozen. Finally, we freeze the gate and fine-tune reconstruction for 1,500 steps using the learned fusion weights. The requested loss and metric definitions are provided in General Response 2.
W1: To further investigate our reliability supervision, we compared the separately trained gate against joint optimization with an auxiliary gate loss (weight 0.3). On the same 12,399 unseen frames, separate training improves correlation with the reliability target from 0.79 to 0.89 and reduces BCE from 0.337 to 0.236. On clean frames, the mean gate increases from 0.835 to 0.970 (target 1.0), while corrupted-frame means remain similar (0.558 vs. 0.548).
These results indicate that separate gate training better preserves the intended visual-reliability signal than joint optimization. Together with the mismatched-audio results (Q1), this supports the motivation for explicitly supervising visual reliability rather than relying solely on reconstruction-driven fusion. We will include this comparison in the revised paper.
W2: We acknowledge that a baseline with fully matched encoders, training data, augmentation, and temporal context would better isolate the architectural contribution. We could not find an available implementation supporting such a controlled comparison.
We therefore extended the evaluation with SPECTRE, trained on 20-frame sequences, and interpolation baselines. On 54 Multiface clips, TAVR4D achieves a mouth-motion correlation of 0.58 under 60-frame occlusions, compared with at most 0.17 for interpolation. Evaluating the same central 10 frames across occlusion lengths, TAVR4D remains at 0.47–0.50, while the best interpolation drops from 0.43 to 0.03.
We also evaluated 14 unseen FaMoS clips with substantial head rotations (30°–60°). TAVR4D achieves 15.2°/16.7° geodesic error at severity 0.6/0.9, compared with 18.9° for the best SLERP baseline at severity 0.6. Our error increases by only 12.6%/23.7% relative to clean input, compared with 4.1–5.5-fold increases for frame-wise baselines. Full interpolation results are provided in EMUT Q1.
These comparisons do not replace a fully matched audio-visual baseline, but provide additional evidence that our robustness gains cannot be explained by simple temporal interpolation alone.
W3: We agree that synchronized audio accounts for most of the benefit under severe mouth occlusion. However, the gate serves a distinct purpose: adapting the fusion weights according to visual reliability. Without the learned gate, fusion uses fixed equal weights (gt=0.5). When video is clean but audio is mismatched, the learned gate improves correlation from 0.61 to 0.82, demonstrating its benefit when visual information should dominate (see EMUT Q2). The additional reliability-supervision experiment in W1 further supports our gate-training strategy.
W4: We will expand Appendix 7.1 with complete loss definitions and implementation details, and Appendix 7.3 with the gate-training and freezing schedule, as clarified in Q3 and General Response 2.