acceptodds
Withdrawn from ICLR 2027

GRAPE: Graduated Routing for Articulated Portrait mesh Estimation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Articulated portrait mesh estimation is fundamental to 3D understanding, avatar generation, and immersive interaction. However, face-centric models suffer from the `floating head` assumption, conflating head pose with global rotation due to the lack of neck kinematics. Conversely, body-centric models lack high-fidelity facial expression capabilities. Furthermore, current methods struggle to disentangle jaw articulation from expression blendshapes, often over-relying on expressions for mouth opening. These limitations make monocular portrait recovery difficult across representation, supervision, and anatomical parameter estimation. To address these limitations, we introduce **GRAPE** (Graduated Routing for Articulated Portrait mesh Estimation). We build a **Portrait Parametric Model (PPM)** with an explicit torso-to-head kinematic chain and a canonical injection step to merge FLAME and the SMPL-X torso. We propose a**Progressive Anatomical Alignment (PAA)** network, which is composed of a pretrained portrait encoder, a graduated-mask router, and coarse-to-fine experts that follow the portrait anatomical prior. We then train this network with multi-source supervision that combines sparse anatomical keypoints, feature distillation, foreground mask constraints, and relative geometry constraints. Experiments show that GRAPE improves portrait mesh recovery quality, pose alignment, and jaw–expression disentanglement over prior methods. We also demonstrate that our method can benefit the downstream tasks of audio-driven talking-head generation and 3D portrait generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Portrait mesh estimation, Talking head, Anatomical disentanglement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Et1FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses monocular articulated portrait mesh estimation. It identifies three gaps in current pipelines: (i) a representation gap — face-centric 3DMMs assume a "floating head" with no torso-rooted kinematic reference, while body-centric models lack FLAME-level facial expression; (ii) a supervision gap — neck/shoulder/torso variables are weakly supervised in-the-wild; (iii) a jaw–expression shortcut — regressors let expression blendshapes absorb mouth opening, leaving the jaw joint nearly static. The authors propose: (a) the Portrait Parametric Model (PPM), which fuses a FLAME-2023 head onto an SMPL-X torso in canonical space via eye-center alignment and neck-ring smoothing, rooting the kinematic chain at the spine; (b) Progressive Anatomical Alignment (PAA), a three-expert (Global/Mid/Local) query-based regressor on a frozen Sapiens-2 encoder, trained with a Graduated-Mask Router that randomly drops deeper-expert supervision so articulation must explain the image before expression is available; (c) multi-source supervision (Sapiens-2 keypoints, Pixel3DMM feature distillation, foreground masks, relative geometry). Experiments report the lowest average 68-point NME across four in-the-wild datasets, the highest jaw–skeletal consistency proxy, head/face vertex errors comparable to TEASER on NoW/Stirling, and downstream gains for DiffPoseTalk and RGBAvatar.

Critical strengths
  1. Well-diagnosed problem with a matched training formulation. The jaw–expression shortcut is precisely characterized, and the Graduated-Mask Router directly targets it: forcing ψ=0 when the L-expert is masked means mouth opening must be explained by jaw articulation. The ablation supports the mechanism (JSR 0.368 flat → 0.710 w/o router → 0.799 full), and Fig. 5's JCR-vs-jaw-pitch time series is convincing evidence that the gain is mechanistic, not incidental.
  2. A usable, reusable artifact. PPM gives the community a simple torso-rooted portrait representation built from standard assets (FLAME + SMPL-X), with demonstrated downstream utility (DiffPoseTalk training-loss reduction, RGBAvatar floating-head artifact removal). This kind of representation fix is often adopted.
Critical weaknesses
  1. Evaluation partially circular on the headline metric. JSR measures correlation between predicted jaw pitch and JCR, but JCR is built from the same nose–chin landmarks that L_nc supervises. A method could score well by overfitting that 2D cue without genuinely better skeletal consistency. An independent validation (e.g., measured jaw excursion from multi-view or EMOCA/SPECTRE-style perceptual judges, or re-animation quality judged by a third-party renderer) would materially strengthen RQ2.
  2. Component attributions are muddy. The ablation shows graduated pseudo-labels (not the router) carry most of the NME gain, and the Nersemble-V2 vertex advantage is confounded by pretraining on that dataset. As a result, the reader cannot cleanly tell how much of the headline improvement is the proposed routing vs. better labels and data.
Questions for authors
  1. For Table 3, Nersemble-V2: were the test identities/sequences disjoint from the stage-1 pretraining split? Please state the exact split.
  2. JSR reuses the L_nc landmark cue. Can you report jaw–skeletal consistency measured against a cue not used in training (e.g., 3D jaw angle from Nersemble multi-view, or mouth-opening from an off-the-shelf audio-visual sync model)?
  3. Table 1: each baseline "uses its own official crop" — GRAPE sees neck/shoulders, which could systematically change the 68-point normalization. How was NME normalized (inter-ocular? bounding-box?), and does the comparison remain fair under a unified normalization?
Reviewer_TdevOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GRAPE proposes a monocular articulated portrait reconstruction framework combining a FLAME head with an SMPL-X torso. It introduces hierarchical regression with graduated masking to reduce entanglement between jaw rotation and facial expression, supported by supervision from multiple sources. Experiments show improved facial alignment and jaw consistency, competitive 3D reconstruction accuracy, and modest improvements in downstream avatar applications.

Critical strengths
  1. Combining a FLAME head with an SMPL-X torso provides a consistent kinematic structure for the head, neck, and shoulders, addressing the floating head limitation of face reconstruction methods.
  2. GRAPE achieves substantial improvements in facial alignment and the reported jaw consistency metric while maintaining competitive 3D reconstruction accuracy. The ablation studies support the effectiveness of the proposed training framework.
Critical weaknesses
  1. The central claim of reducing entanglement between jaw rotation and facial expression is evaluated primarily through JSR, which measures correlation with a nose to chin distance ratio derived from landmarks also used in training supervision. Although this provides useful evidence of jaw consistency, it does not directly establish anatomically accurate jaw rotation.
  2. The proposed representation combines existing FLAME and SMPL-X models, while the evaluation lacks direct measurements of head and neck rotation accuracy. Consequently, the extent to which the method improves articulated motion, beyond reconstruction and joint alignment, remains insufficiently established.
Questions for authors

Q1. Beyond the JSR metric and qualitative comparisons, what evidence supports the claim that GRAPE recovers anatomically accurate jaw rotation rather than simply improving agreement with the supervised nose to chin distance?

Q2. Could the authors clarify how the ground truth for MPJPE is obtained and whether the evaluation provides evidence of accurate head and neck articulation?

Reviewer_11W7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes GRAPE that regresses upper-body portrait model(PPM) from a single image. A frozen Sapiens2 encoder feeds three query-based experts for global, articulation and expression parameters. Graduated-Mask Router randomly masks the articulation and expression branches such that the jaw explains mouth opening, not by expression during training. Results show that GRAPE improves jaw-landmark correlation(JSR) on HDTF. Authors also extend expreriments to head avatar animation.

Critical strengths
  1. Head-jaw tracking ambiguity is a well-defined problem. Head pose relative to a torso-anchored neck and jaw-driven mouth opening are essential in animation pipelines, where face-only regressor may not be able to achieve none of them. Figure 5. & 9. also provide failure modes well. Although being the major strength, the rating has been made considering weaknesses together.

  2. Simple routing design has a clear improvement within the paper’s own metric (0.710 -> 0.799) as probability decreases from (1.0 -> 0.5) (Table 5) while keeping NME and MVE unchanged.

Critical weaknesses
  1. Contributions are limited to jaw-head pose disentanglement whose measurement and training signal are circular. PPM is essentially PEAR’s EHM-s while not clarified. Multi-source supervision is more like a combination of standard losses. Although admitted, circular signal for L_nc and JSR may lead to polluted evaluation. Additionally, individual components’ effect of Progressive Anatomical Alignment (PAA) is even not sufficiently analyzed that are responsible for marginal head/face vertex error improvements from previous SOTA methods. This weakness affects my rating the most.

  2. Unexplained discrepancy between submission and supplementary material(code) includes PEAR distillation. RelativeGeoLoss runs PEAR online as the L_rg(line 911) target which is supposed to be supervised with offline Graduated-Refinement Pipeline(line 276). Also use_pear_body_gt=True signal replaces the torso pose, body shape and head-scale targets with PEAR predictions(loss.py, paa_pipeline.py) which may affect MPJPE in Table 4. Misalignment between submission and code for router mechanism confuses the interpretation of Table 5 as well. This is supportive to the rating decision.

Questions for authors
  1. Are there major differences between PPM from EHM-s? If there are substantial differences, could ‘PPM -> EHM replacement’ ablation be reported?

  2. Would authors justify implementation choice from unexplained difference including PEAR prediction distillation and replacement, router mechanism and L_nc?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.