acceptodds
Withdrawn from ICLR 2027

VideoFace: Learning a Discrete Motion Space for Sparse-to-Dense Facial Motion Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing facial motion from sparse visual keyframes requires visual estimation under appearance changes and temporal completion between observations. Independently estimated states and subsequent interpolation do not explicitly model how an expression develops, including its relative intensity, regional coordination, and timing. We present VideoFace, a framework that combines visual conditioning from a pretrained vision-language model with a learned discrete motion prior. Motivated by the potential of semantic representations to support robustness across appearances, the model uses keyframe images, their temporal indices, and motion history to predict a complete motion-token sequence of a specified length. A dual-branch tokenizer connects ARKit and FLAME through shared finite scalar quantization, assigning one token per frame. Self- and cross-reconstruction preserve motion content, while window-level code matching and frame-level alignment encourage correspondence between the two representations. After tokenizer training, we freeze its parameters and fine-tune the vision-language model for autoregressive token prediction; either decoder then produces the corresponding continuous trajectory. We also construct MAMC-Face, comprising 10 hours of facial motion at 30 Hz rendered across 65 avatars under moving camera views. Its paired visual–motion supervision supports learning complete motion trajectories from sparse visual observations across varied appearances and viewpoints.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Facial Motion Reconstruction, Vision-Language Models, Motion Tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qazyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses reconstructing dense facial motion (ARKit/FLAME, 30 Hz) from sparse keyframes. A dual-branch FSQ tokenizer maps ARKit and FLAME into one shared code space, and a fine-tuned Qwen3.5-4B predicts the motion tokens autoregressively from the keyframe images and their frame indices. The authors also build MAMC-Face, 10 hours of ARKit-driven motion rendered on 65 avatars with moving cameras. VideoFace outperforms five reconstruction baselines on this dataset in both dense and sparse settings.

Critical strengths

The sparse-to-dense setting is well motivated. Fig. 1 makes a good point: low per-frame error does not guarantee correct onset/hold/release timing or coordination across facial regions.

MAMC-Face pairs rendered images with the exact controls that drove them, and the same motion is rendered on multiple avatars and viewpoints. This is useful for studying appearance-invariant motion estimation, and could benefit the community if released.

Critical weaknesses
  1. [Dataset] I found it difficult to understand how MAMC-Face is split. As far as I can tell, the paper only states an 8/1/1 hours split by motion. I could not find how the 65 avatars and the source performers are distributed across the splits. This matters because the paper claims robustness across appearances: if test avatars or performers also appear in training, the results may not show generalization to unseen faces or motion styles. I may be misreading this, but the numbers also seem inconsistent: the text mentions 10 h at 30 Hz (≈1.08M frames), while Table 1 reports 864K motion frames, which corresponds to 8 h (perhaps only the training set?). Finally, all results are on the rendered dataset; results on real videos would make the claims more convincing.

  2. [Sparse setting and baselines] I could not find the values of M and N or how keyframes are sampled in the sparse experiments (Table 3). It is also unclear how the baselines, most of which are single-image methods, are evaluated in this setting: are missing frames filled by interpolation, or are metrics computed only on keyframes? This matters both for fairness and for interpreting Table 3. Relatedly, the introduction argues against “estimate + interpolate” pipelines, but it is not clear whether this pipeline is effectively what the baselines represent. Finally, a non-VLM in-betweening model on the same tokens would help clarify what the pretrained VLM itself contributes, since the current ablation (Table 4) only compares output formats.

  3. [Benefit of the dual-branch design for generation] The VLM is trained only on ARKit tokens (Eq. 5), and all quantitative results are in ARKit space. The paper argues that aligning with FLAME makes the token space less identity-dependent and therefore helps the VLM (L428–431), but this is supported only by t-SNE plots and a code-overlap number. An ablation in Table 3 comparing an ARKit-only tokenizer, the dual-branch tokenizer without CMCM/MMS, and the full version would make this claim much stronger. The tokenizer’s own reconstruction error, which bounds the achievable accuracy, is also not reported.

  4. [Evaluation] The dual encoder used for R@1/MMD is not described (architecture, training data). Unlike the standard evaluators used for these metrics in text-to-motion work, its reliability cannot be judged, which matters since R@1 shows the largest gaps. The text-based baseline in Table 4 also lacks a citation and implementation details, and its MSE values come from “separate archived runs.” Minor suggestions: since timing is the paper’s main motivation (Fig. 1), a metric that reflects it more directly (e.g., velocity/acceleration error) would strengthen the claims, and variance across runs would help assess the margins.

  5. [Novelty] The main components (FSQ, the CMCM/MMS objectives of Liu et al. (2022), and LLM-based motion-token generation) already exist, so in my view the technical contribution lies mostly in combining them for the sparse-keyframe setting, together with the dataset. Since the task is essentially keyframe-conditioned in-betweening, a discussion of motion in-betweening work (e.g., Harvey et al., 2020) would also help position the paper.

  6. Minor

  • Please report the loss weights and temperatures (λ_c, λ_CMCM, λ_MMS, τ_c, τ_d, τ_f, μ), the inference cost compared with the baselines, and how SPECTRE/TEASER/OFER were retrained for ARKit output.

  • More details on dataset construction would help: how ARKit parameters were extracted from the performances and “refined” (L286), the source and license of the avatars, and consent for the performances, especially if the dataset is to be released.

  • Regional Mean Error” covers only the jaw/lip coefficients (App. A.1). Stating this in the main text, or adding eye/brow regions, would help.

  • “NR” in Table 1 is not defined.

Questions for authors
  1. How are the 65 avatars and the source performers distributed across the train/val/test splits? Does the 864K figure in Table 1 refer to the full dataset or only the training set?

  2. In the sparse setting, what are M and N, and how are keyframes sampled? How are the baselines evaluated: are missing frames interpolated, or are metrics computed only on keyframes?

  3. How does a non-VLM model with the same tokenizer and keyframe features perform?

  4. Does the dual-branch tokenizer improve ARKit generation compared with an ARKit-only tokenizer? What is the tokenizer’s reconstruction error?

  5. How was the evaluation dual encoder for R@1/MMD trained, and on what data?

  6. Are there any results on real videos?

Reviewer_RkghOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies sparse-to-dense facial motion reconstruction: given M keyframes from a video, their frame indices, and a target length N, predict the full N-frame motion. VideoFace has three parts. (1) A dual-branch ARKit–FLAME tokenizer uses a shared FSQ codebook with one token per frame and is trained with self- and cross-reconstruction plus window-level (CMCM) and frame-level (MMS) alignment. (2) A Qwen3.5-4B VLM, fine-tuned with LoRA, autoregressively predicts the motion-token sequence from the keyframes, their indices, and previous tokens. (3) MAMC-Face contains 10 hours of human-derived ARKit motion at 30 Hz rendered on 65 avatars under moving cameras, with the driving controls kept as labels. On MAMC-Face, VideoFace reports large gains over five reconstruction baselines in both dense and sparse settings.

Critical strengths
  1. The problem is well framed. The paper separates matching observed states from preserving dynamics (amplitude, onset–hold–release, regional coordination), and conditioning on frame indices and target length makes dense reconstruction the special case M = N.
  2. MAMC-Face keeps the controls that actually drove each rendering instead of re-estimating labels, and renders the same motion across avatars and viewpoints. This is a sound way to obtain paired supervision.
  3. One token per frame gives the VLM a fixed-length output aligned with the timeline. Table 4 shows why this matters: with sparse input, numerical-text prediction is valid only 10.5% of the time, versus 100% for motion tokens.
Critical weaknesses

I recommend rejection. The central claim, that VideoFace recovers temporally faithful motion between sparse observations, cannot be verified in the current submission for the reasons in W1–W3.

W1. The sparse protocol is unspecified, so Table 3 is hard to interpret. The paper does not give N, the keyframe interval or M/N ratio, how keyframes are selected, or whether training and testing use the same scheme. None of the baselines is designed to fill in missing frames, and the paper does not say how they produce N-frame trajectories from M keyframes: interpolation, holding the last value, or scoring only the observed frames. The same question applies to their smoothness ratings in the sparse user study. One number makes this pressing: MediaPipe's error drops in the sparse setting (MAE 0.1007 → 0.0963, MSE 0.0301 → 0.0252), which suggests the two settings may be scored on different frames. There are also no errors split into observed vs. unobserved frames and no results across sparsity levels, although recovering the unobserved intervals is the stated contribution.

W2. Task-matched baselines are missing. Two comparisons are essential: (a) the strongest per-frame estimator applied at the keyframes, followed by linear or spline interpolation, and (b) a non-VLM temporal predictor trained on the same data with the same visual features, both under the same keyframe sampling. The first tests whether learned completion beats a simple rule; the second tests whether the VLM is needed at all. The "Text-based + linear interpolation" row in Table 4 ablates the output format and does not substitute for either. Training conditions are also uneven: SPECTRE, TEASER, and OFER are retrained on MAMC-Face (without details), but the paper does not say whether MediaPipe and SemanticFace receive any adaptation, or which tool produced the ground-truth ARKit labels.

W3. The sequence-level metrics cannot test dynamics. The paper argues (Fig. 1) that low per-frame error does not guarantee correct dynamics. Yet FFD (Eq. 9) compares per-coefficient standard deviations, and FD (Eq. 10) treats frames as independent samples. Both are invariant to frame order: shuffling the predicted frames leaves them unchanged, so they cannot measure timing, onset, release, or ordering. Order-sensitive measures are needed, such as velocity and acceleration error or peak-timing error on the unobserved intervals. Separately, R@1 and MMD rely on an undescribed "fixed evaluation dual encoder", and several baselines score below the 1/32 chance level on R@1 (e.g., MediaPipe 0.0090 and SemanticFace 0.0156, sparse). The encoder's architecture, its training data, and whether it was trained on MAMC-Face or with VideoFace's tokenizer need to be stated.

W4. The dual-branch tokenizer is not validated downstream. The VLM is trained only on ARKit tokens (Eq. 5), and every quantitative result is in ARKit space, so the FLAME path that motivates the dual-branch design is never evaluated. There are no quantitative errors for the four decoding paths, no downstream comparison with the ARKit-only tokenizer shown in Fig. 4(b), and no ablation of cross-reconstruction, CMCM, or MMS on final reconstruction quality. Fig. 4(a)–(b) show that alignment changes the latent distribution, not that it improves results. Simpler ways to support both formats exist, such as multi-head prediction across annotation conventions (UniTalker) or template-based blendshape-to-FLAME retargeting (EmoTalk). ARKit-only generation followed by retargeting is the natural comparison.

W5. Generalization is untested. All quantitative results are on the authors' own synthetic data, split only by motion duration (8/1/1 h). The paper does not say whether test avatars, source performers, or camera trajectories are held out, so the gains may reflect fitting a familiar rendering distribution rather than appearance-invariant understanding. The robustness across appearances claimed in the abstract is not tested, and there is no quantitative evaluation on real video.

Positioning. The novelty lies mainly in the combination of existing ideas. FaceGPT already places 3DMM parameters in a VLM's token space, and MotionGPT already evaluates motion in-betweening with discrete tokens. For the dataset, Lu et al. (2024) used a similar pipeline (blendshapes from real video, retargeted to rendered avatars) for AU detection and should be compared in Table 1. This makes W2 the key test of whether the combination brings concrete gains.

Questions for authors

Answers to Q1–Q4 are the most likely to raise my score.

  1. What are N, the keyframe interval, the selection strategy, and the train/test sampling scheme? How do the baselines produce full-length sequences in Table 3, are dense and sparse results scored on the same frames, and why does MediaPipe improve under sparse input?
  2. Can you report errors on observed vs. unobserved frames, and results at several sparsity levels?
  3. How does VideoFace compare with keyframe estimation plus interpolation and with a non-VLM temporal predictor under identical inputs?
  4. Can you add at least one order-sensitive dynamics metric, and describe the evaluation encoder behind R@1 and MMD?
  5. How does the dual-branch tokenizer compare downstream with an ARKit-only tokenizer, and how does the FLAME output compare with ARKit-only generation followed by retargeting?
  6. Are test avatars, performers, and camera trajectories disjoint from training? Is there any evaluation on real video?

If the rebuttal specifies the sparse protocol, shows that Table 3 compares like with like, and VideoFace remains clearly ahead of the baselines in Q3 under identical inputs, I would raise my score.

Reviewer_WZrHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

VideoFace reconstructs a complete 30 Hz facial motion trajectory from sparse keyframe images. A dual-branch causal tokenizer maps paired ARKit and FLAME trajectories into a shared FSQ vocabulary (1,000 codes, one token per frame), trained with self- and cross-reconstruction plus two alignment losses (window-level code matching and frame-level matching). The tokens are added to Qwen3.5-4B, which is fine-tuned with LoRA to predict the full token sequence from keyframe images, their frame indices, and the target length; either decoder then produces continuous ARKit or FLAME parameters. The paper also builds MAMC-Face, 10 hours of ARKit motion rendered on 65 avatars under moving cameras. On the MAMC-Face test split, VideoFace reports the lowest errors against MediaPipe, SemanticFace, and retrained SPECTRE, TEASER, and OFER, in both dense and sparse settings, and a 10-person user study.

Critical strengths
  1. A practical shared interface for two facial representations. One token vocabulary that decodes to either ARKit controls or FLAME parameters is convenient for downstream animation, and the alignment losses increase shared code usage (0.258 to 0.675).
  2. Tokens are better than numeric text output for this task. Table 4 shows that direct numeric prediction collapses under sparse input (10.5% valid outputs), while token prediction stays valid.
Critical weaknesses
  1. The evidence does not show that the method works on captured video. All quantitative evaluation is on the authors' own synthetic renderings (MAMC-Face), so the margins in Tables 2–3 measure in-domain fit, and the paper does not state whether test avatars overlap with training avatars (Q3). Recent facial capture methods such as SMIRK (CVPR 2024), SEREP (ICCV 2025), and SHeaP (ICCV 2025) already recover detailed expressions on real video and are evaluated on captured data. Here, the qualitative results show mainly subtle mouth and brow changes on a single frontal target avatar (Fig. 3, supplementary video), which does not suggest a gain over these methods. The end use, driving a digital face from video, is also served by portrait animation models such as X-NeMo (ICLR 2025) and HunyuanPortrait (CVPR 2025). These solve a different task (pixel generation rather than parameter recovery), so I do not count the missing comparison against the paper, but the paper should state when a parametric output is preferable.

  2. The method combines existing components, and their contributions are not analyzed. VideoFace combines a pretrained VLM (Qwen3.5-4B with LoRA), an FSQ motion tokenizer with two alignment losses, and rendered avatar data. The data pipeline is not documented: the tool used to extract ARKit motion from human performances, the trajectory refinement, and the avatar rig and renderer are unspecified, and the dataset release is not stated (Q8). The only component ablation is Table 4. There is no ablation of the VLM backbone or size, of the alignment losses on the final VLM error, or of the number of keyframes, no ARKit-only vs. dual-tokenizer comparison under the VLM, no FLAME-output metric, and no "per-keyframe estimation + interpolation" baseline, although the sparse setting samples keyframes every 2–4 frames according to the released code. The technical and academic contribution is therefore unclear.

Questions for authors
  1. (Decision-relevant) What keyframe spacing and count are used in Table 3, and how do the baselines produce non-keyframe outputs? What does SemanticFace or TEASER obtain with linear or spline interpolation at the same spacing?
  2. (Clarification) When the goal is to drive an avatar from video, how does VideoFace compare in quality and cost with recent portrait animation models such as X-NeMo (ICLR 2025) or HunyuanPortrait (CVPR 2025)? Which component (VLM, tokenizer and alignment losses, rendered data) is responsible for the gains in Tables 2–3?
  3. (Decision-relevant) Do test avatars overlap with training avatars? Can you report results on captured video?
  4. (Clarification) Tables 2–3 have no floor or ceiling, so the absolute numbers are hard to read. Please report (a) the tokenizer's own round-trip error (encode and decode ground-truth trajectories), which bounds what the VLM can reach; (b) a trivial predictor (dataset mean, or holding the nearest keyframe estimate); and (c) R@1 of ground-truth motion under the evaluation encoder, which bounds R@1. The 10-person user study should also report confidence intervals.
  5. (Clarification) The dual-branch tokenizer is a main contribution, but VLM results are reported for ARKit only, and the evidence for shared motion semantics is t-SNE plots (Fig. 4a–b) plus one code-overlap number. Can you report FLAME-output metrics and a comparison with an ARKit-only tokenizer under the same VLM?
  6. (Clarification) How does the shared ARKit–FLAME tokenizer differ from prior unified multi-format facial animation, e.g., UniTalker (ECCV 2024, one model with ARKit and FLAME output heads), and from FLAME motion tokenizers with autoregressive generation such as MMHead (ACM MM 2024)?
  7. (Clarification) What is the "fixed evaluation dual encoder" used for R@1 and MMD, and what data was it trained on?
  8. (Clarification) What is the source of the human performances and the 65 avatars, under what consent and license, and will MAMC-Face be released?

Results on captured video (Q3) that keep a clear margin over an interpolation baseline (Q1) would move my rating to 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.