- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper addresses reconstructing dense facial motion (ARKit/FLAME, 30 Hz) from sparse keyframes. A dual-branch FSQ tokenizer maps ARKit and FLAME into one shared code space, and a fine-tuned Qwen3.5-4B predicts the motion tokens autoregressively from the keyframe images and their frame indices. The authors also build MAMC-Face, 10 hours of ARKit-driven motion rendered on 65 avatars with moving cameras. VideoFace outperforms five reconstruction baselines on this dataset in both dense and sparse settings.
Critical strengths
The sparse-to-dense setting is well motivated. Fig. 1 makes a good point: low per-frame error does not guarantee correct onset/hold/release timing or coordination across facial regions.
MAMC-Face pairs rendered images with the exact controls that drove them, and the same motion is rendered on multiple avatars and viewpoints. This is useful for studying appearance-invariant motion estimation, and could benefit the community if released.
Critical weaknesses
-
[Dataset] I found it difficult to understand how MAMC-Face is split. As far as I can tell, the paper only states an 8/1/1 hours split by motion. I could not find how the 65 avatars and the source performers are distributed across the splits. This matters because the paper claims robustness across appearances: if test avatars or performers also appear in training, the results may not show generalization to unseen faces or motion styles. I may be misreading this, but the numbers also seem inconsistent: the text mentions 10 h at 30 Hz (≈1.08M frames), while Table 1 reports 864K motion frames, which corresponds to 8 h (perhaps only the training set?). Finally, all results are on the rendered dataset; results on real videos would make the claims more convincing.
-
[Sparse setting and baselines] I could not find the values of M and N or how keyframes are sampled in the sparse experiments (Table 3). It is also unclear how the baselines, most of which are single-image methods, are evaluated in this setting: are missing frames filled by interpolation, or are metrics computed only on keyframes? This matters both for fairness and for interpreting Table 3. Relatedly, the introduction argues against “estimate + interpolate” pipelines, but it is not clear whether this pipeline is effectively what the baselines represent. Finally, a non-VLM in-betweening model on the same tokens would help clarify what the pretrained VLM itself contributes, since the current ablation (Table 4) only compares output formats.
-
[Benefit of the dual-branch design for generation] The VLM is trained only on ARKit tokens (Eq. 5), and all quantitative results are in ARKit space. The paper argues that aligning with FLAME makes the token space less identity-dependent and therefore helps the VLM (L428–431), but this is supported only by t-SNE plots and a code-overlap number. An ablation in Table 3 comparing an ARKit-only tokenizer, the dual-branch tokenizer without CMCM/MMS, and the full version would make this claim much stronger. The tokenizer’s own reconstruction error, which bounds the achievable accuracy, is also not reported.
-
[Evaluation] The dual encoder used for R@1/MMD is not described (architecture, training data). Unlike the standard evaluators used for these metrics in text-to-motion work, its reliability cannot be judged, which matters since R@1 shows the largest gaps. The text-based baseline in Table 4 also lacks a citation and implementation details, and its MSE values come from “separate archived runs.” Minor suggestions: since timing is the paper’s main motivation (Fig. 1), a metric that reflects it more directly (e.g., velocity/acceleration error) would strengthen the claims, and variance across runs would help assess the margins.
-
[Novelty] The main components (FSQ, the CMCM/MMS objectives of Liu et al. (2022), and LLM-based motion-token generation) already exist, so in my view the technical contribution lies mostly in combining them for the sparse-keyframe setting, together with the dataset. Since the task is essentially keyframe-conditioned in-betweening, a discussion of motion in-betweening work (e.g., Harvey et al., 2020) would also help position the paper.
-
Minor
-
Please report the loss weights and temperatures (λ_c, λ_CMCM, λ_MMS, τ_c, τ_d, τ_f, μ), the inference cost compared with the baselines, and how SPECTRE/TEASER/OFER were retrained for ARKit output.
-
More details on dataset construction would help: how ARKit parameters were extracted from the performances and “refined” (L286), the source and license of the avatars, and consent for the performances, especially if the dataset is to be released.
-
Regional Mean Error” covers only the jaw/lip coefficients (App. A.1). Stating this in the main text, or adding eye/brow regions, would help.
-
“NR” in Table 1 is not defined.
Questions for authors
-
How are the 65 avatars and the source performers distributed across the train/val/test splits? Does the 864K figure in Table 1 refer to the full dataset or only the training set?
-
In the sparse setting, what are M and N, and how are keyframes sampled? How are the baselines evaluated: are missing frames interpolated, or are metrics computed only on keyframes?
-
How does a non-VLM model with the same tokenizer and keyframe features perform?
-
Does the dual-branch tokenizer improve ARKit generation compared with an ARKit-only tokenizer? What is the tokenizer’s reconstruction error?
-
How was the evaluation dual encoder for R@1/MMD trained, and on what data?
-
Are there any results on real videos?