acceptodds
Under review as a conference paper at ICLR 2027

TAVR4D: Temporal Audio-Visual Transformers for Reliable 4D Head Reconstruction

Abstract

Monocular 3D head reconstruction is accurate when visual evidence is reliable, but real videos often contain occlusions, illumination changes, motion blur, and other image degradations. Under these conditions, frame-wise estimators can interpret corrupted observations as motion, leading to large errors and jitter in pose and expression prediction. Temporal smoothing can reduce jitter but does not recover motion when the current observation is missing or unreliable. We introduce TAVR4D, a long-context multimodal model for robust 4D head reconstruction from degraded video. We formulate reconstruction as a temporal inference problem in which different motion components benefit from different complementary sources of information. The model uses long-range visual context to estimate head rotation, neck pose and translation when individual frames are unreliable. It also uses synchronized speech to help estimate facial expression and jaw motion. A learned visual reliability gate controls the balance between visual and audio features. We evaluate TAVR4D under 14 visual degradation conditions, including real-hand and full-face occlusions as well as dropped frames, which are held out as explicit image-space corruption classes during training. Across conditions shared by all baselines, TAVR4D remains close to its clean-input reconstruction accuracy, while remaining competitive on clean input. Our results show that combining long-range temporal inference with reliability-gated audio-visual fusion improves motion recovery and reconstruction robustness under the evaluated visual degradations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Head Reconstruction, 4D Head reconstruction, FLAME, 3D Morphable Models, Multimodal Reconstruction, Audio-Visual Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_iXYrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TAVR4D for 4D head reconstruction under visual degradation. The method uses separate bidirectional temporal branches for global head motion and facial articulation, with synchronized audio providing additional information for expression and jaw prediction. A learned visual-reliability gate controls audio-visual fusion and selects a frame for static identity estimation with MICA.

Critical strengths
  • The paper presents a well-motivated problem and effectively decomposes the task. It tackles head reconstruction when individual frames are unreliable, sensibly separating global head motion from speech-related articulation. This approach assigns clear, complementary roles to temporal visual context and synchronized audio.
  • The evaluation considers head rotation and mouth-motion tracking, not just surface error. Results show significant improvements despite severe visual degradation. Removing audio greatly reduces mouth-motion correlation, highlighting its role in recovering hidden articulation.
Critical weaknesses
  • Limited methodological novelty. Relative to AVFace as discussed in the manuscript, the main changes are longer temporal context, separate motion branches, and reliability-gated fusion. These appear to be incremental architectural modifications rather than a substantial methodological advance. The paper does not establish why its specific reliability supervision offers an advantage over conventional learned fusion.
  • Most baselines operate on individual visual frames, whereas TAVR4D uses synchronized audio and 100-frame bidirectional context. These comparisons demonstrate a system-level advantage but do not distinguish the proposed design from the benefit of additional information. An audio-visual temporal baseline with matching encoders, training data, augmentation, and context length is needed.
  • The gate has modest gains and inconsistent ablation results. Under severe lower-face occlusion, correlation is 0.56 for the full model, 0.54 without gating, and 0.21 without audio, suggesting that audio provides most of the benefit. Gating helps more clearly with mismatched audio, but its clean-input results are inconsistent. Table 5 reports 0.86 without gating, whereas Table 9 reports 0.76, with the full model at 0.84 in both. This discrepancy reverses the conclusion about the gate’s benefit and must be resolved.
  • Appendix 7.1 lists loss groups and weights without defining several individual terms, including the mouth-closure, geometry, and motion-range losses. The expression branch is trained before the gate, but the gating values used during this stage are not specified.
Questions for authors
  • What is the main technical advance over prior audio-visual reconstruction beyond longer context, separate prediction branches, and scalar gating?
  • Why does the no-gate variant obtain a clean-input correlation of 0.86 in Table 5 but 0.76 in Table 9? Do these results use different checkpoints or evaluation settings?
  • What gating values are used during training of the expression branch, considering the gate is trained later? Is the reconstruction network updated afterward to incorporate the learned gate? Please provide precise definitions for mouth-closure, geometry, motion-range, and motion-variation losses.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Q1: While our visual and audio encoders are pretrained, our contribution lies in how their information is used for different motion components. Long-range visual context supports global head motion recovery, while synchronized speech provides complementary information for facial articulation. Rather than treating both as a single reconstruction task, TAVR4D uses separate temporal branches and a supervised visual reliability gate for adaptive audio-visual fusion and identity-frame selection. Our ablations support this design: restricting visual context increases rotation error from approximately 6° to 12.6° (Table 11), while removing audio reduces mouth-motion correlation from 0.56 to 0.21 under severe occlusion. The gate provides a smaller improvement under occlusion (0.56 vs. 0.54), but a clearer benefit when visual input is reliable and audio is mismatched (0.82 vs. 0.61).

Q2 – Gate ablation discrepancy. We reran the evaluation using the same checkpoints and evaluation setup and confirmed that the 0.86 reported for w/o gate in Table 5 is a transcription error. The correct value is 0.76, consistent with Table 9; 0.86 belongs to w/o audio. We will correct Table 5 accordingly.

Q3: Gate training and loss definitions. Expression reconstruction initially uses fixed audio-visual fusion (gt=0.5g_t=0.5). We then train the reliability gate separately using DINOv2 clean/corrupted feature similarity, with reconstruction frozen. Finally, we freeze the gate and fine-tune reconstruction for 1,500 steps using the learned fusion weights. The requested loss and metric definitions are provided in General Response 2.

W1: To further investigate our reliability supervision, we compared the separately trained gate against joint optimization with an auxiliary gate loss (weight 0.3). On the same 12,399 unseen frames, separate training improves correlation with the reliability target from 0.79 to 0.89 and reduces BCE from 0.337 to 0.236. On clean frames, the mean gate increases from 0.835 to 0.970 (target 1.0), while corrupted-frame means remain similar (0.558 vs. 0.548).

These results indicate that separate gate training better preserves the intended visual-reliability signal than joint optimization. Together with the mismatched-audio results (Q1), this supports the motivation for explicitly supervising visual reliability rather than relying solely on reconstruction-driven fusion. We will include this comparison in the revised paper.

W2: We acknowledge that a baseline with fully matched encoders, training data, augmentation, and temporal context would better isolate the architectural contribution. We could not find an available implementation supporting such a controlled comparison.

We therefore extended the evaluation with SPECTRE, trained on 20-frame sequences, and interpolation baselines. On 54 Multiface clips, TAVR4D achieves a mouth-motion correlation of 0.58 under 60-frame occlusions, compared with at most 0.17 for interpolation. Evaluating the same central 10 frames across occlusion lengths, TAVR4D remains at 0.47–0.50, while the best interpolation drops from 0.43 to 0.03.

We also evaluated 14 unseen FaMoS clips with substantial head rotations (30°–60°). TAVR4D achieves 15.2°/16.7° geodesic error at severity 0.6/0.9, compared with 18.9° for the best SLERP baseline at severity 0.6. Our error increases by only 12.6%/23.7% relative to clean input, compared with 4.1–5.5-fold increases for frame-wise baselines. Full interpolation results are provided in EMUT Q1.

These comparisons do not replace a fully matched audio-visual baseline, but provide additional evidence that our robustness gains cannot be explained by simple temporal interpolation alone.

W3: We agree that synchronized audio accounts for most of the benefit under severe mouth occlusion. However, the gate serves a distinct purpose: adapting the fusion weights according to visual reliability. Without the learned gate, fusion uses fixed equal weights (gt=0.5g_t=0.5). When video is clean but audio is mismatched, the learned gate improves correlation from 0.61 to 0.82, demonstrating its benefit when visual information should dominate (see EMUT Q2). The additional reliability-supervision experiment in W1 further supports our gate-training strategy.

W4: We will expand Appendix 7.1 with complete loss definitions and implementation details, and Appendix 7.3 with the gate-training and freezing schedule, as clarified in Q3 and General Response 2.

Thank you for the reply and additional experiments. The correction of the gate ablation and the explanation of the training procedure address my corresponding concerns, and the interpolation results provide useful additional evidence. However, my main concern about the methodological contribution remains unresolved. The added comparisons do not provide an audio-visual temporal baseline with matched inputs and training conditions. Moreover, the new gate experiment compares two supervised training strategies and reports agreement with the reliability target, without establishing downstream reconstruction benefits over conventional learned fusion. These results therefore do not sufficiently isolate the benefits of the proposed design from those of additional audio and temporal information. I lean to keep my original rating.

Reviewer_rT8fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TAVR4D regresses FLAME head pose, expression, and jaw parameters from degraded monocular video. Frozen DINOv2 features feed two bidirectional temporal Transformers over a 100-frame clip: a pose branch that uses only visual context, and an expression/jaw branch that fuses visual and Whisper audio features through a learned scalar visual-reliability gate. The gate is supervised by the DINOv2 cosine distance between clean and corrupted versions of a frame, and it also selects the frame used for MICA identity estimation. The model is evaluated on 54 Multiface clips under 14 degradation conditions (including held-out real-hand occlusion, full-face occlusion, and dropped frames) and on 14 FaMoS clips with large head motion. Vertex error stays at 1.13–1.17 mm across conditions, and mouth-motion correlation under severe lower-face occlusion is 0.56 vs. at most 0.04 for frame-wise baselines.

Critical strengths
  1. The design is decomposed and each part is tested directly. The context-length experiment (Table 11, Fig. 6) shows that full-clip context keeps head-rotation error flat (6.03° to 6.37°) as occlusion grows to 60 frames, while a ±10-frame window degrades to 12.57°, with a bootstrap CI. The audio ablations (Tables 4–5) show that audio supplies mouth motion under occlusion (0.56 vs. 0.21 without audio) and that wrong audio hurts (0.13), and corruption training reduces jitter (Table 6). This is convincing evidence for the paper's central hypothesis.
  2. Honest reporting of where the method loses. The paper shows that it is worse on clean large head turns (13.5° vs. 5.4° for DECA), slightly worse on clean mouth error than the strongest image-based methods (Table 1: 2.034 vs. 1.979 mm for Pixel3DMM), and offline only. Held-out corruption types are used for evaluation.
Critical weaknesses
  1. The cheapest temporal baseline is missing. In the protocol, each corruption covers at most 75 of 100 frames and leaves clean frames before and after. The natural baseline is a frame-wise estimator (e.g., SMIRK or DECA) whose outputs on corrupted frames are replaced by interpolation from the clean neighbours (possibly using the same gate to detect them). The abstract argues that temporal smoothing cannot recover missing motion, but this is not tested. For mouth motion, an audio-driven animation method (e.g., FaceFormer- or EMOTE-style) is the equivalent control.
  2. Evaluation scope and confounds. Quantitative results use only studio-captured data (54 Multiface clips, 14 FaMoS clips) with synthetic degradations. In-the-wild video appears only as qualitative examples (Fig. 3 and the supplementary video). Prior work usually adds an in-the-wild evaluation, for example emotion recognition from the reconstruction plus a perceptual study (EMOCA, CVPR 2022) or lip-reading-based metrics and user studies (SPECTRE, CVPRW 2023). In addition, the vertex-error gap in Tables 2 and 7 may partly reflect identity handling, since TAVR4D fixes identity from a clean high-gate frame and it is unclear whether the clean-first-frame identity used for baselines in Sec. 4.2 also applies to Tables 2 and 7 (see Q4).
Questions for authors
  1. (Clarification) What is the main contribution beyond combining strong pretrained components (frozen DINOv2 features and reliability target, Whisper, MICA identity, and Pixel3DMM pseudo-labels)? An ablation that swaps the visual encoder or the pseudo-label source would show how much of the robustness comes from these components.
  2. (Decision-relevant) All baselines are frame-wise. What do they obtain when corrupted frames are replaced by interpolation from clean neighbours, and how does TAVR4D compare with temporal FLAME video methods such as SPECTRE (CVPRW 2023) or AU-aware dynamic reconstruction (Kuang et al., WACV 2024)?
  3. (Decision-relevant) Can you add a quantitative in-the-wild evaluation on naturally degraded videos, such as emotion or lip-reading metrics from the reconstruction, or a user study?
  4. (Clarification) In Tables 2, 7, and 8, how were the baselines' identity parameters set? Can you report baselines with identity fixed from a clean frame, and a constant mean-face predictor on the heavily occluded frames?
  5. (Clarification) The expression branch is trained before the gate exists. What fusion weight was used during that training stage, and how many subjects do the 54 Multiface clips cover?

If the interpolation baseline in Q2 closes most of the gap, I would lower my rating to 2. If TAVR4D stays clearly ahead and Q3 adds in-the-wild evidence, I would keep my rating.

2 comments
AuthorsOpenReview ↗

Official Comment by Authors

Q1: While we build on existing pretrained encoders, our contribution lies in separating long-range visual inference for head motion from audio-visual inference for facial articulation, with a supervised visual-reliability gate controlling fusion and identity-frame selection. Our ablations support this decomposition: removing audio reduces mouth-motion correlation from 0.56 to 0.21 under severe occlusion, while restricting visual context increases rotation error from approximately 6° to 12.6° (Table 11). The gate provides a modest improvement under severe occlusion (0.56 vs. 0.54), where audio already supplies most of the missing articulation information. However, its contribution becomes more pronounced when visual evidence is reliable. On clean videos, removing audio slightly improves correlation (0.86 vs. 0.84), while fixed audio-visual fusion reduces it to 0.76 (Table 9). The learned gate mitigates this interference, improving reconstruction correlation from 0.76 to 0.84 by favoring reliable visual features. Under mismatched audio, it further improves correlation from 0.61 to 0.82. Thus, the gate enables adaptive fusion rather than indiscriminately relying on audio. We have not conducted additional visual-encoder or pseudo-label-source ablations.

Q2: On 14 unseen FaMoS clips with substantial head rotations (30°–60°), TAVR4D achieves 15.2° geodesic rotation error at severity 0.6, compared with 18.9° for the best SLERP control and 35.2° for the best hold baseline. For mouth motion, we evaluated 54 Multiface clips with different occlusion lengths, reporting Pearson correlation over occluded frames. At an occlusion length of 60 frames, TAVR4D achieves 0.58, compared with at most 0.17 for interpolation. The full comparison is provided in EMUT Q1. These results demonstrate that simple interpolation does not explain our recovery under prolonged occlusions.

Q3: We additionally evaluated ten unseen VoxCeleb2-test videos with naturally occurring lip occlusions. A MediaPipe-based pipeline selects windows with at least 30% lip coverage. Since 3D ground truth is unavailable, we report SyncNet lip-synchronization scores (LSE-D lower, LSE-C higher is better), rather than geometric reconstruction error. TAVR4D achieves the best scores on both metrics, providing initial quantitative evidence beyond composited occlusions.

Method (10 occluded windows)LSE-DLSE-C
TAVR4D11.693.62
SPECTRE13.561.39
SMIRK13.081.46
DECA13.461.34

Q4: As specified in Sec. 4.2, each baseline's identity is estimated from its own first clean frame and held fixed throughout the sequence, including Tables 2, 7, and 8. Only expression and pose predictions vary across frames. Thus, the reported comparisons already use fixed clean-frame identities for all baselines. We have not additionally evaluated a constant mean-face predictor.

Q5: Before gate training, fusion uses g = 0.5. We then train the gate separately, insert it into the expression reconstruction network. To expression reconstruction network to adapt, we fine-tune the reconstruction network 1500 steps with the gate frozen. Multiface evaluation uses 54 clips from 8 subjects.

W1: We agree that comparisons with simple motion-completion methods are important. We therefore added SPECTRE, trained on 20-frame sequences, together with interpolation baselines. These experiments directly evaluate whether the improvements can be explained by temporal interpolation rather than learned motion recovery (EMUT Q1).

W2: We acknowledge that the main quantitative benchmarks use studio-captured data with controlled degradations. To extend the evaluation beyond synthetic occlusions, we additionally tested naturally occluded videos (Q3). Regarding identity estimation, all baselines already use identity parameters fixed from their own clean frame, including Tables 2, 7, and 8 (Q4).

Reviewer_rT8fOpenReview ↗

Response to rebuttal

Thank you for the additional experiments and clarifications. My assessment after the rebuttal:

Resolved

  • Q2 (interpolation baseline): for occlusions of 20 frames or more, frame-wise estimators with interpolation do not close the gap (0.58 vs. at most 0.17 at L = 60 on Multiface; 15.2° vs. 18.9° for the best SLERP control on FaMoS at severity 0.6). This meets the condition stated in my review, so I keep my rating.
  • Q4: thank you for confirming that each baseline fixes identity from its own clean first frame, including Tables 2, 7 and 8.
  • Q5: answered (g = 0.5, then separate gate training, then 1,500 fine-tuning steps with the gate frozen; 54 clips from 8 subjects). The Table 5 correction (0.76 for w/o gate) also resolves the inconsistency raised by the other reviewers.

Partially resolved

  • Q2, attribution: the mouth-motion comparison sets TAVR4D (video + audio) against visual-only interpolation, so it does not separate long-range visual context from audio. The one-sided-context table in your reply to EMUT points the same way: before-only, after-only and both-sides context give nearly identical correlation (differences of at most 0.01). An audio-only mouth baseline (FaceFormer- or EMOTE-style, or an audio-only variant of TAVR4D) would show how much the visual pathway adds on occluded frames.
  • Short occlusions: at L = 10 the margin over DECA + interpolation is small (0.47 vs. 0.43). Please state from which occlusion length long-range context clearly pays off, and add a confidence interval to the 14-clip FaMoS comparison, as done for Table 11.
  • Q3 (in-the-wild): the VoxCeleb2 result (LSE-D 11.69 vs. 13.08 to 13.56) is useful initial evidence, but it covers 10 windows and a lip-sync metric only, which may favor the only audio-conditioned method in the comparison. Please report how many identities the windows cover, add a confidence interval, and state in the paper that quantitative in-the-wild evidence is limited to lip sync.

Open (not decision-changing)

  • Q1: the encoder / pseudo-label-source ablation, the comparison with Kuang et al. (WACV 2024), and the constant mean-face predictor were not addressed; please list them as limitations.
  • The PDF has not been revised yet. Please upload a revision with the promised changes (Table 5 correction, App. 7.1 loss and metric definitions, App. 7.3 gate schedule, interpolation and one-sided-context tables).

My rating stays at 3.

Reviewer_TWQdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing monocular 3D face reconstruction methods often operate frame by frame. When the mouth is occluded, or the video suffers from low light, blur, or other degradations, these methods may interpret corrupted visual observations as changes in pose or expression, producing inaccurate predictions and temporal jitter. Temporal smoothing can reduce jitter, but it cannot supply the missing motion information.

The authors argue that different motion components benefit from different sources of information. Head rotation, translation, and neck pose can be inferred from long-range visual context, while facial expressions and jaw motion benefit from both visual observations and synchronized speech. Accordingly, the model uses two separate branches. The pose branch processes frozen DINOv2 visual features through a temporal Transformer to predict head motion. The expression branch combines DINOv2 visual features with frozen Whisper-large-v3 audio features to predict FLAME expression parameters and jaw rotation. A visual reliability gate adjusts the fusion weights, increasing the contribution of audio when visual evidence becomes less reliable. The model also selects the most reliable frame in the sequence and uses MICA to estimate static identity parameters, which remain fixed throughout the sequence.

The main evaluation covers 14 visual degradation conditions on 54 Multiface clips, with an additional evaluation of head-motion recovery on 14 FaMoS clips containing large head movements.

Critical strengths

1. The design is well motivated, and targeted experiments support the proposed division of information sources. The method assigns different roles to long-range visual context and synchronized speech according to the motion being recovered. The occlusion-duration experiment shows that local-context predictions deteriorate once the occlusion extends beyond the available window, while the full-context model remains relatively stable. For mouth motion, removing audio reduces the correlation under severe lower-face occlusion from 0.56 to 0.21, demonstrating that synchronized speech provides useful complementary information.

2. The experiments examine several factors directly related to the proposed design. Beyond reconstruction accuracy, the authors investigate occlusion duration, temporal context, audio conditioning, and reliability gating. Occlusions composited from real hand images, full-face occlusions, and dropped frames are held out as explicit corruption types during training, providing some evidence of generalization across degradation types. The mismatched-audio experiment also examines whether incorrect audio can distort reconstruction when the visual input is clear.

3. The model retains competitive reconstruction accuracy on clean inputs while improving robustness under degradation. On the main Multiface evaluation, the model achieves a clean-input whole-face vertex error of 1.140 mm, indicating that improved robustness is accompanied by competitive overall geometric accuracy. However, this result should be interpreted in light of differences in identity estimation, and it does not establish equally accurate recovery of every local detail or dynamic expression.

Critical weaknesses

1. The evaluation does not sufficiently rule out simple interpolation as an explanation for part of the performance. Multiface contains relatively limited head motion, and the evaluation consistently preserves clean frames before and after the degraded interval. Under these conditions, the surrounding poses may already provide enough information to estimate intermediate states through simple interpolation. Although the paper includes context ablations and additional baselines, such as holding the most recent reliable pose or interpolating between reliable poses before and after the occlusion, would help establish the improvement over simple motion-completion strategies.

2. Performance across degradation severity levels is not sufficiently characterized. Only lower-face and full-face occlusions are evaluated at multiple severity levels. The remaining degradations are primarily tested at severity 0.9. Performance curves from mild to severe degradation, including corresponding baseline results, would clarify whether the method provides consistent benefits across severity levels or whether its advantages mainly emerge when extreme degradation causes frame-wise baselines to fail.

3. The evaluation of facial dynamics remains concentrated on a limited set of motion dimensions. The paper provides relatively direct evidence for recovering head rotation and mouth opening, but offers less detailed validation of other expression dimensions, local 3D deformations, and their temporal evolution. For example, mouth-opening trajectories do not fully capture lip shape, mouth-corner movement, or other facial actions. More detailed region-level evaluations of dynamic geometry would strengthen the evidence for facial-motion recovery.

Questions for authors

1. Tables 5 and 9 report inconsistent results. Under clean video and clean audio, the mouth-motion correlation of the model without the gate is 0.86 in Table 5 but 0.76 in Table 9. The appendix text also uses 0.76.

2. The caption of Figure 3 reverses the actual layout. The caption on page 7 states that each column represents a clip and each row represents a method. The figure instead shows one clip per row and one method per column.

3. The individual loss terms in Appendix 7.1 are not fully defined. The appendix provides the loss groups and their weights but does not explicitly define every constituent loss. Please provide the mathematical definitions or include the relevant references and specify any modifications.

4. The FaMoS training and test split needs clarification. Please specify whether the split is performed by original sequence, clip, or other division methods. It would also be helpful to clarify whether training and test clips share identities, come from temporally adjacent segments, or contain overlapping frames.

5. The meaning of each degradation severity level should be specified. Please provide the corruption-generation parameters corresponding to severity levels 0.3, 0.6, and 0.9 for each applicable degradation. In particular, the mapping between occlusion severity and occluded area or other relevant attributes should be made explicit to improve interpretability and reproducibility.

6. Generalization to other objects and naturally occurring occlusions remains unclear. The current hand-occlusion evaluation uses composited real hand images. Additional evaluations involving other occluding objects and naturally occurring occlusions in real videos would help determine whether the model generalizes beyond the current synthetic corruption protocol.

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Q1: We reran the evaluation using the same checkpoints and evaluation setup. The 0.86 reported for w/o gate in Table 5 is a transcription error. The correct value is 0.76, consistent with Table 9; 0.86 belongs to w/o audio. We will correct Table 5 accordingly.

Q2: Thank you for pointing this out. We will correct it.

Q3: Loss and metric definitions are provided in General Response 2.

Q4: Expression training uses 780 VoxCeleb2 videos from 780 identities and 580 FaMoS clips from 70 additional identities, totaling 850 training identities. The head-pose branch is trained exclusively on VoxCeleb2. For head-pose evaluation, we use 14 FaMoS clips from 10 additional identities, disjoint from all training identities.

Q5: The following parameters are taken from our corruption-generation code. Crop coverage does not necessarily correspond to the fraction of the visible face occluded. Corruptions are applied within 100-frame clips, with bursts affecting at most 75 frames.

CorruptionParameter0.30.60.9
Lower-faceCrop height42.50%50%57.50%
Burst (frames)40–7251–7561–75
Full-faceCrop area30%60%90%
Burst (clip)47%59%71%
Real handWidth of crop27%36%45%
Burst (frames)40–7251–7561–75
Low lightGamma1.62.22.8
Noise std (0–255)11.819.627.4
Dropped framesProbability0.250.40.55
Low resolutionDownsampling3×5×7×
JPEGQuality29198
Motion blurKernel (px)71113
ExposureGain±38%±56%±74%
Camera shakeTranslation std2.80%4.60%6.40%
Rotation std2.8°4.6°6.4°

Q6 – Naturally occluded videos: We evaluated ten unseen VoxCeleb2-test videos with naturally occurring lip occlusions using SyncNet lip-sync scores (LSE-D lower, LSE-C higher is better). TAVR4D performs best on both metrics.

Method (10 occluded windows)LSE-DLSE-C
TAVR4D11.693.62
SPECTRE13.561.39
SMIRK13.081.46
DECA13.461.34

W1: We added SPECTRE (trained on 20-frame sequences), interpolation, and hold baselines. On FaMoS, TAVR4D achieves 15.2° geodesic rotation error at severity 0.6, compared with 18.9° for the best SLERP control. On 54 Multiface clips, we report Pearson mouth-motion correlation over occluded frames:

MethodL = 10L = 20L = 40L = 60
TAVR4D0.470.500.520.58
SPECTRE+Interpolation0.410.240.260.16
SMIRK + Interpolation0.360.210.210.16
DECA + Interpolation0.430.230.280.17
SPECTRE + Hold0.190.080.100.11
SMIRK + Hold0.190.090.140.20
DECA + Hold0.140.160.160.16

At L=60, the best interpolation reaches only 0.17. On the same central ten frames across all lengths, TAVR4D remains at 0.47–0.50, while the best interpolation baseline drops from 0.43 to 0.03.

W2 – We extended the evaluation to severity levels 0.3 and 0.6 for low light, real hand occlusion, and low resolution, complementing Table 3(b). We report Pearson mouth-opening correlation over affected frames (higher is better). Best baseline refers to the highest-performing baseline among Table 3(b).

DegradationMethod0.30.6
Low lightTAVR4D0.780.67
Best baseline0.580.20
Real handTAVR4D0.790.77
Best baseline0.580.42
Low resolutionTAVR4D0.850.84
Best baseline0.860.86

TAVR4D demonstrates stronger robustness under increasing low-light and hand-occlusion severity, while remaining competitive under low resolution. At severity 0.9, its correlation remains at 0.84, compared with 0.76 for the best baseline (Table 8). Our corruption-aware training aims to preserve stable motion estimates across degradation levels.

W3 – Facial dynamics beyond mouth opening: We additionally evaluated mouth width (corner-to-corner distance) and mouth shape (opening/width), using Pearson correlation over affected frames:

ConditionMethodOpeningWidthsShape
Lower face 0.9TAVR4D0.560.370.54
SPECTRE0.030.000.03
SMIRK0.040.040.09
DECA0.020.070.04
Real-hand 0.9TAVR4D0.740.410.65
SPECTRE0.280.150.26
SMIRK0.360.260.34
DECA0.240.190.27
Reviewer_EMUTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces TAVR4D for dynamic head reconstruction under visual degradation. The model uses long-range visual context to estimate global head motion and reliability-gated audio-visual features to predict facial expression and jaw motion. A static identity estimate is shared across the sequence. Evaluation under 14 visual degradation conditions, together with additional large-head-motion experiments, shows improved robustness and mouth-motion recovery compared with frame-wise methods. The ablations indicate that synchronized audio provides substantial complementary information, while the additional benefit of the reliability gate depends on the input condition.

Critical strengths
  1. Clear improvements under severe visual degradation. Under severe lower-face occlusion, the paper reports a mouth-motion correlation of 0.56, compared with approximately 0.02–0.04 for the listed frame-wise baselines.

  2. Evaluation extends beyond static reconstruction accuracy. Mouth-motion correlation, motion amplitude, and large-head-rotation errors help assess whether the model recovers motion rather than merely producing smooth or nearly static predictions.

  3. Targeted temporal-context experiments. Appendix 7.9.2 evaluates identical central frames while varying surrounding occlusion duration and provides clip-level bootstrap confidence intervals, supporting the contribution of distant visual observations.

  4. Important limitations are acknowledged. The paper discusses weaker performance on clean large-motion sequences, sensitivity to mismatched audio, and the offline nature of bidirectional inference.

Critical weaknesses
  1. The advantage over reasonable temporal baselines is not fully established. The short-context and frame-wise variants in Table 11 are obtained by restricting the context of a model trained on 100-frame sequences only at inference time. This intervention both removes information and changes the context distribution. It demonstrates that the model uses distant observations, but does not replace comparisons with a properly trained short-context model or a simple pose-interpolation baseline. This distinction matters because Multiface contains limited rigid head motion.

  2. The gate ablations contain inconsistent results that affect contribution assessment. For clean video and clean audio, the model without the gate achieves a correlation of 0.86 in Table 5 but 0.76 in Table 9, while the full model achieves 0.84 in both. These values imply different conclusions about the gate on clean inputs. Moreover, under severe occlusion, removing audio reduces correlation from 0.56 to 0.21, whereas removing the gate reduces it only to 0.54. The overall robustness gain therefore cannot primarily be attributed to the gate, and the precise fusion operation used in the no-gate variant needs clarification.

  3. The robustness evidence is concentrated on short clips with clean context on both sides. Corruptions affect at most 75 of 100 frames and preserve clean observations before and after the corrupted interval. The “real-hand” condition also uses composited hand occlusions. Furthermore, the 14 degradation conditions share the same 54 Multiface clips and should not be interpreted as independent datasets. The test identity distribution and the separation between FaMoS training and evaluation data are insufficiently specified.

  4. Several training and evaluation details remain underspecified. Appendix 7.1 lists loss groups and weights but does not fully define the mouth-closure, geometry, and motion-range terms. The gate is trained after the reconstruction model, leaving unclear which gate values are used while training the reconstruction branch. Precise definitions of excess acceleration and the motion-amplitude ratio are also needed.

Questions for authors
  1. How does the method compare with pose interpolation or smoothing using the same visible frames, and with a model trained for short-context inference? Does the long-occlusion advantage persist on sequences with substantial head motion?

  2. Why do Tables 5 and 9 report different clean-input results for the no-gate model? Please specify the checkpoints, fusion operations, and evaluation settings, and compare against a reasonable fixed-weight fusion baseline.

  3. How does performance change when occlusion occurs at the beginning or end of a clip, leaving reliable context on only one side? Please also clarify test identity counts, relationships among clips, and the FaMoS data split.

  4. Please provide exact definitions of the key losses, excess acceleration, and motion-amplitude ratio, together with the gate values and freezing schedule used during each training stage.

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Q1: To measure this, we evaluated different interpolation baselines on 54 Multiface clips. Results are in the table below. Pearson correlation is used for evaluation. Independently, we evaluated 14 unseen FaMoS clips with large head rotations (30°–60°). TAVR4D achieves 15.2° at 0.6 severity of occlusion, compared with 18.9° for the best SLERP control. The best hold-baseline yields 35.2° geodesic rotation error.

MethodL = 10L = 20L = 40L = 60
TAVR4D0.470.500.520.58
SPECTRE+Interpolation0.410.240.260.16
SMIRK + Interpolation0.360.210.210.16
DECA + Interpolation0.430.230.280.17
SPECTRE + Hold0.190.080.100.11
SMIRK + Hold0.190.090.140.20
DECA + Hold0.140.160.160.16

Q2: We reran the evaluation using the same checkpoints and evaluation setup to verify the results. The 0.86 reported for w/o gate in Table 5 is a transcription error. The correct value is 0.76, consistent with Table 9; 0.86 belongs to w/o audio. We will correct Table 5 accordingly.

The fusion is ft=gtvt+(1−gt)Proja(at)f_t = g_t v_t + (1-g_t)\mathrm{Proj}_a(a_t). The w/o gate variant is already trained with a fixed gt=0.5g_t = 0.5 for the results in Table 5 and Table 9.

Q3: We tested clean context on both sides of the occlusion, before only, and after only, on the same 54 Multiface clips with a mixture of occlusion settings, 0.9/0.6/0.3 on full face uniformly sampled:

Pearson CorrelationL = 20L = 40L = 60
Both sides0.5010.5250.577
Before only0.5040.5160.571
After only0.5070.5240.574

For expression training, we used 780 VoxCeleb2 videos from 780 identities and 580 FaMoS clips from 70 identities. The pose branch is trained only on VoxCeleb2. Expression evaluation uses 54 Multiface clips from 8 subjects; head-pose evaluation uses 14 FaMoS clips from 10 subjects, disjoint from the FaMoS training subjects.

Q4: Loss and metric definitions are provided in General Response 2. Expression reconstruction starts with fixed audio–visual fusion (gt=0.5g_t=0.5). We then train the gate using DINOv2 clean/corrupted feature similarity (Eq. (2)) and soft-label BCE (Eq. (3)), with reconstruction frozen. Finally, we freeze the gate and fine-tune reconstruction for 1,500 steps using learned fusion weights.

W1: We agree that restricting context at inference does not replace a separately trained short-context model. Therefore, we compare against SPECTRE and interpolation baselines to provide temporal alternatives and demonstrate the limitations of simple interpolation under prolonged occlusion (Q1).

W2: We agree that audio accounts for most of the benefit under severe mouth occlusion. However, its contribution depends on visual reliability. On clean videos, removing audio slightly improves correlation (0.86 vs. 0.84), whereas fixed fusion without the gate reduces it to 0.76 (Table 9). The learned gate recovers correlation to 0.84, demonstrating its ability to mitigate unnecessary audio influence when visual evidence is reliable. This is further supported by our mismatched-audio experiment, where the gate improves correlation from 0.61 to 0.82(Table 5).

W3: The 14 degradation conditions reuse the same 54 Multiface clips, rather than separate dataset. Test and training identity details are answered in Q3, and will be added to the paper. We also tested naturally occluded videos, see answer to TWQd Q6.

W4: We will expand Appendix 7.1 with complete loss/metric definitions and Appendix 7.3 with the gate-training and freezing schedule (Q4; General Response 2).

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers(1/2):

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are pleased that all four reviewers found our work interesting or useful. The reviewers acknowledge our well-motivated approach and principled decomposition of motion components (TWQd, iXYr), strong robustness under severe visual degradations (EMUT, TWQd, rT8f, iXYr), and convincing experimental evidence supporting our central hypothesis (EMUT, TWQd, rT8f). Furthermore, the reviewers appreciate our targeted ablation studies (EMUT, TWQd, rT8f), evaluation of motion recovery beyond static reconstruction accuracy (EMUT, TWQd, iXYr), and competitive performance on clean inputs (EMUT, TWQd).

Following the initial reviews, we have conducted additional experiments and provided detailed responses to each reviewer's questions and concerns. In particular, we have added comparisons against interpolation baselines, evaluated naturally occurring occlusions, and performed additional ablation studies. We hope that these results address the remaining concerns and help to further strengthen our paper.

AuthorsOpenReview ↗

General response to all reviewers(2/2):

We would like to clarify the definitions of our losses and evaluation metrics.

Let αt=∥θt∥2\alpha_t = \|\theta_t\|_2 denote the jaw-opening magnitude, where θt\theta_t is the jaw axis-angle rotation.

Mouth-closure loss: We introduce this loss to emphasize accurate reconstruction of near-closed mouth configurations:

Lclo=∑twtκt(αt^−αt)2σα2∑twtκtL_{\mathrm{clo}} = \frac{\sum_t w_t \kappa_t (\hat{\alpha_t} - \alpha_t)^2}{\sigma_\alpha^2 \sum_t w_t \kappa_t}

Here, κt\kappa_t assigns higher weights to near-closed configurations.

Geometry loss: We measure the geodesic distance between the predicted and ground-truth jaw rotations:

Lgeo=Etdg(R(θt^),R(θt))L_{\mathrm{geo}} = \mathbb{E}_t d_g(R(\hat{\theta_t}), R(\theta_t))

Here, dgd_g denotes the geodesic distance between two rotations in SO(3)SO(3).

Motion-range loss: This loss penalizes an underestimated jaw-motion range:

Lrng=[max⁡(0,ρ(α)−ρ(α^))max⁡(ρ(α),10−3)]2L_{\mathrm{rng}} = \left[\frac{\max(0,\rho(\alpha)-\rho(\hat{\alpha}))}{\max(\rho(\alpha),10^{-3})}\right]^2

Here, ρ\rho denotes the within-sequence difference between the 95th and 5th percentiles.

Motion-variation loss: Similarly, we penalize underestimated temporal variation:

Lvar=1D∑d=1D[max⁡(0,sd(y)−sd(y^))]2L_{\mathrm{var}} = \frac{1}{D}\sum_{d=1}^{D}\left[\max(0,s_d(y)-s_d(\hat{y}))\right]^2

Here, sds_d denotes the within-sequence standard deviation of parameter dd.

Evaluation metrics: We define the motion-amplitude ratio as std⁡(h^)/std⁡(h)\operatorname{std}(\hat{h})/\operatorname{std}(h), and excess acceleration as RMS⁡(Δ2h^)/RMS⁡(Δ2h)\operatorname{RMS}(\Delta^2\hat{h})/\operatorname{RMS}(\Delta^2h), where hh denotes the ground-truth mouth-opening trajectory over the evaluated frames. For both metrics, a value of 1 indicates matched magnitude. Since all reported excess-acceleration ratios exceed 1, lower values are better within the evaluated range, as indicated in Table 6.

The remaining reconstruction and temporal losses follow the parameter supervision and mean–deviation decomposition described in Appendix 7.1 and Eqs. (9)–(11). We will provide their complete mathematical definitions and implementation details in the revised Appendix 7.1.

AuthorsOpenReview ↗

Rebuttal Overview

We would like to thank all the reviewers for taking the time to read our work and for giving us valuable suggestions and feedback. Their comments have helped us strengthen both the experimental evaluation and the presentation of our work. Based on their comments we have uploaded a revised version of the paper. The main changes we made during the rebuttal are as follows:

  • Additional baselines: We tested SPECTRE, a temporal reconstruction method, as well as interpolation and hold baselines for frame-wise methods. For mouth motion, the correlation reaches 0.58 compared to 0.17 for the best interpolation baseline. For head motion, our method achieves an error of 15.2° versus 18.9° for the best interpolation baseline. (Sec. 4.4–4.5, Table 5, Appendices 7.8 and 7.9.2)

  • Additional robustness evaluations: We used MediaPipe to select 10 naturally occluded VoxCeleb2 video windows and compared methods using lip-sync metrics, where our method performed better. We also extended the evaluation to mouth width and shape, and additional degradation severity levels. (Sec. 4.3, Appendix)

  • Ablations: We corrected the no-gate result and added a comparison with joint gate training. We also evaluated one-sided temporal context and the contribution of long-range visual context to head-motion recovery. (Sec. 4.5, Table 6, Appendices 7.9.1–7.9.2)

  • Reproducibility: We explained the loss and metric definitions, training details, dataset splits, and corruption parameters. (Sec. 3.1, Appendices 7.1 and 7.3–7.6)

We would also like to emphasize that our contribution is more than combining audio and temporal information. We separate global head motion from articulation and model when to use audio and when to use video. Our experiments show that long-range visual context supports head-motion recovery under prolonged occlusions, while synchronized audio provides complementary information for facial articulation when visual observations are unreliable.

The closest work to our approach is AVFace, however, their implementation is not publicly available. While AVFace also uses audio-visual input, it does not employ an explicit visual reliability gate as proposed in our work. Our fixed-weight ablation uses the audio and visual inputs showing that the learned gate improves the mouth-motion correlation from 0.76 to 0.84 on clean inputs and from 0.61 to 0.82 with mismatched audio, on clean frames. We also compared joint gate training to look into this design choice and found that separate training better preserves the intended visual reliability signal, as measured by agreement with the reliability target.

These revisions increased the main paper by 1 page (from 9 to 10 pages) and the appendix by 3 pages (from 6 to 9 pages).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.