acceptodds
Withdrawn from ICLR 2027

FaceGID: Facial Mesh Expression Retargeting with Geometry-image Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D facial expression retargeting remains challenging due to the irregular structure of mesh data, the scarcity of large-scale 3D facial datasets, and the limitations of relying on fixed templates or reference meshes. In this work, we formulate this task as a 2D conditional generation problem using geometry images as the 3D representation. This allows us to fine-tune a pretrained 2D image generative model on geometry images and leverage priors learned from large-scale 2D data for 3D generation. Given a source facial mesh and a single reference image, our method transfers the reference expression while preserving the source identity and capturing expression-related appearance changes. We further introduce a training-free guidance strategy that enables controllable expression strength at inference time. Experiments demonstrate effective expression retargeting and good generalization across different facial meshes and reference images.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Expression retargeting, Generative model, Geometry images, 3D representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fKMgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FaceGID is a diffusion-based method for facial expression retargeting. Specifically, given facial geometry associated with a particular identity and a reference image depicting a target expression, FaceGID first converts the facial geometry into 2D images that encode geometric information, allowing it to leverage a 2D diffusion model to reconstruct 3D geometry. It then fine-tunes a LoRA adapter based on FLUX.2 to perform expression retargeting. The trained LoRA generates geometry images with the retargeted expression, which are subsequently converted back into 3D geometry. In addition, the authors propose a training-free method to further refine the expression retargeting results during inference.

Critical strengths
  1. The paper is well written and enables readers to quickly understand the core ideas.
  2. The overall framework is reasonably designed, and reformulating facial expression retargeting as a 2D conditional generation task is insightful.
Critical weaknesses
  1. The robustness of the method appears limited. Converting facial geometry into 2D geometry images allows FaceGID to leverage the powerful priors of 2D diffusion models, but it may also compromise the robustness of geometric reconstruction. Since 2D diffusion models generally have limited control over fine-grained details, small errors may not pose significant problems for ordinary images. However, for geometry images, even minor inaccuracies can lead to spatial misalignment of mesh vertices. Such errors may originate from multiple sources, including denoising errors in the diffusion model and reconstruction errors introduced by the VAE.
  2. The method cannot fully replace existing mature solutions. The authors demonstrate that FaceGID can produce more realistic expressions than expression-basis replacement methods based on 3D Morphable Models (3DMMs). However, traditional methods retain significant advantages in terms of efficiency and support for diverse input styles, making them difficult to replace in practice. This limits the practical impact of FaceGID's contributions.
  3. Overall, I consider FaceGID a reasonably designed method supported by a relatively comprehensive set of experiments. My main concern is that it addresses a relatively narrow and long-standing problem for which mature solutions already exist. Furthermore, the proposed approach primarily involves transferring and combining existing techniques from closely related areas, while still being unable to fully replace established solutions. I am therefore uncertain whether its contributions meet the acceptance threshold of a top-tier conference. At this stage, I lean toward rejection, although my final decision will also depend on the other reviewers' assessments and the authors' rebuttal.
Questions for authors

My primary concerns are not about the technical details of the method, which I believe are explained reasonably clearly in the paper. Instead, I would like the authors to elaborate on the robustness of the proposed approach and its practical use cases. Based on the current presentation, I find it difficult to see how FaceGID could replace expression-basis replacement techniques in practical applications such as video games and VR, where predefined expression templates can enable facial expression retargeting that is both fast and reliable.

Reviewer_W38WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a 3D facial expression relocalization framework called FaceGID, which represents facial meshes as geometric images and adapts a pre-trained 2D generative model to this representation. Given a source facial mesh and a reference RGB image, the method aims to transfer the reference expression while preserving the source identity and generating a texture consistent with that expression.

The framework consists of three main components: (1) a geometric image representation combining position, normals, and texture maps; (2) domain adaptation of a pre-trained VAE, followed by fine-tuning of a correction-flow Transformer based on LoRA; (3) an expression-guided strategy during the inference stage based on CPEM expression features.

Experiments conducted on the FaceScape dataset—which includes a topologically unified mesh and raw facial scan data—demonstrate that this method outperforms both CPEM and SMIRK in terms of identity recognition and facial expression metrics for rendered images. Additionally, supplementary experiments explore the impact of expression-guidance intensity.

Critical strengths
  1. By representing facial geometry and appearance through geometric images, this method provides a practical interface for pre-trained image generation models. Its potential value lies in integrating facial geometric transformations and appearance synthesis into a unified generative pipeline.

  2. Whether applied to topologically consistent meshes or raw facial scan data, the method outperforms CPEM and SMIRK. Qualitative examples are encouraging, and the guidance strength analysis in Table 3 indicates a quantifiable trade-off between facial expression consistency and identity preservation. Furthermore, the method utilizes an on-the-fly guidance mechanism that requires no additional model training, a feature that is highly attractive for practical applications.

Critical weaknesses
  1. The accuracy of the 3D model itself was not directly measured, so it cannot be proven that the performance improvement is reflected at the geometric level.

The core of this paper is the implementation of facial expression transfer for 3D facial meshes; however, all current quantitative metrics are calculated based on rendered 2D images, and no direct measurements were taken of the accuracy of the 3D mesh’s vertex positions or surface geometry.

It is currently impossible to distinguish whether the improvement in metrics stems from increased accuracy in the 3D geometry itself or is merely due to rendered lighting and visual effects appearing closer to the target. Furthermore, the magnitude of improvement in existing metrics is small, and without actual geometric error measurements or statistical significance tests, it is impossible to prove that the 3D geometry itself has achieved substantial progress.

  1. The design advantages of the paper’s proposed “selective freezing VAE” lack sufficient experimental evidence to support them.

The paper claims that freezing only a subset of VAE parameters—rather than performing full fine-tuning—preserves the latent space characteristics of the pre-trained model; however, this assertion lacks sufficient evidence to support it.

On the one hand, the KL divergence in the formula does not explicitly specify the exact nature of the two distributions being compared, and merely “freezing a portion of the network layers” does not guarantee that the latent space is truly compatible with the original pre-trained model; On the other hand, the existing comparative experiments simultaneously altered both the number of input channels and the training strategy, making it impossible to isolate the effects attributable solely to “selective freezing.”

Furthermore, the paper did not conduct comparative experiments with a “9-channel VAE with full-parameter fine-tuning,” so it cannot prove that this design actually provides additional gains compared to standard 9-channel joint modeling.

Questions for authors
  1. In Tables 1 and 2, are the surface normals used for the textureless “(mesh)” evaluation calculated based on the reconstructed mesh vertices, or are they taken from the generated normal map? Similarly, how are the normals used for differentiable rendering in Equation (5) determined? Are there any constraints to ensure consistency between the generated position map and the normal map?

  2. The current evaluation relies on image-based metrics, even for the “(mesh)” results. Could the authors provide direct 3D geometric metrics—such as chamfer distance or surface normal error—using test pairs with known target facial expression meshes? Additionally, how many identity samples and test pairs were used in Tables 1 and 2, respectively? Please report variances or confidence intervals to assess the reliability of the reported improvements.

  3. In Equation (1), which distributions does the KL divergence compare? How does this term maintain compatibility with the pre-trained latent space? Could the authors compare the proposed “selective freezing” strategy with full fine-tuning of a 9-channel VAE under the same training conditions? Furthermore, a comparison with and without the KL term would help clarify its contribution.

  4. The paper claims to support grids with different triangular subdivisions, but the preprocessing steps described in Appendices A.3 and A.4 involve corresponding topologically uniform grids or registration. For a never-seen facial grid, what preprocessing steps are required? Does inference require a registered reference grid, or can the method handle new grids independently?

  5. Given that the guidance mechanism uses CPEM facial features, how does it perform on extreme or asymmetrical expressions that CPEM may not fully characterize? Can the authors provide a more detailed evaluation for such cases? Additionally, what are the inference times and GPU memory consumption with and without expression guidance enabled?

  6. Could the authors clarify how CPEM and SMIRK were applied in the various comparisons? Furthermore, how were the neutral expression images in Equation (9) obtained, particularly for the generated results?

Reviewer_yUNuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FaceGID takes a textured source 3D face mesh and a single photo of another person's expression, and generates a new face mesh and texture that reflect the reference expression while preserving the source identity. It maps the face surface to a square domain, represents position, normal, and texture as a 9-channel geometry image, and performs conditional generation with an adapted VAE and FLUX.2 fine-tuned with LoRA. During inference, it decodes and renders the predicted clean latent and corrects the flow velocity to reduce the distance between the CPEM expression features of the render and those of the reference photo.

In two evaluations—FaceScape TU meshes with rendered references, and FaceScape raw scans with real reference photos—the paper reports lower expression and identity distances than CPEM and SMIRK. The improvement is particularly large in identity preservation, and the guidance-strength experiment shows a trade-off between better expression matching and identity preservation. The final output is not a deformation of the input mesh with its vertex connectivity preserved, but a textured 3D face reconstructed from the geometry image.

Critical strengths
  1. A coherent system that combines photo-based expression conditioning with the source 3D face information. Through the geometry image, faces with different vertex counts and connectivity are handled in a common representation, and expression-dependent shape and appearance are generated jointly. The input is not restricted to a fixed FLAME representation, and applying the method not only to TU meshes but also to raw scans is meaningful. The multi-view renderings in the supplementary material help confirm that the results are actual 3D surfaces. Explicitly stating the topology and alignment constraints on the input surface and the limitation in generating the eye and mouth interiors is also useful for understanding the scope of the evaluation.

  2. Quantitative evidence of the identity-preservation advantage over the compared methods and of the effect of guidance. In the values reported in Tables 1–2, relative to the better baseline for each metric, identity distance decreases by about 20.6–39.5%, and expression distance also improves. These figures are reductions in feature distance on rendered faces. Separating training, validation, and test subjects in the TU evaluation is meaningful for judging performance on unseen identities. In Table 3, as the guidance strength increases, expression distance first decreases and then increases again, while identity distance keeps increasing. This shows not only the effect of the correction but also the need to choose an appropriate strength and the trade-off with identity preservation. Since guidance uses CPEM while expression evaluation uses OpenFace, the reported evaluation values are not simply the same feature loss that is optimized.

Critical weaknesses
  1. The contribution of the pretrained 2D generative prior—the core motivation—is not verified in isolation. Combining geometry images with pretrained image generative models also appears in prior work cited in the manuscript, such as GIMDiffusion, and correcting the flow with a differentiable loss on the clean estimate is related to FlowChef. This does not mean these works already solve the same overall task as FaceGID, but to judge the contribution of the proposed system, it is important to show experimentally what advantage this combination, tailored to facial expression transfer, provides. The main quantitative ablation currently focuses on guidance strength and does not compare the effect of pretrained initialization under the same representation and generator architecture. The qualitative before/after comparison of VAE adaptation in the appendix also does not isolate the gain from generator pretraining. In particular, the no-guidance condition in Table 3 (η=0\eta=0) already outperforms both baselines on all four metrics of the TU evaluation, so the benefit of the overall system should be distinguished from the additional effect of guidance. From the current results, it is difficult to assess the individual contributions of the geometry-image representation, the model architecture/capacity, and pretraining—especially the gain from pretraining when 3D data are limited.

  2. The evaluation supporting the system's practical benefit is limited. The input requires a textured face mesh and preprocessing, and registration may be needed for semantic consistency. The output does not preserve the original vertex connectivity or rig, and it cannot generate the eye and mouth interiors. Generating static 3D expression assets is a valid task, but it is not sufficiently demonstrated in which production tasks the benefit justifies these preparation and post-processing costs. Inference time and memory usage are also not reported, which makes it difficult to evaluate the system's value in terms of both result quality and computational cost.

    The quantitative comparison is limited to CPEM and SMIRK. Under real-photo conditioning, expression distance decreases by about 3.8% in the textured evaluation and about 3.2% in the untextured evaluation relative to the better baseline for each metric. Without repeated runs or uncertainty statistics, it is difficult to judge the stability of these small differences. I acknowledge the large advantage in identity preservation, but differences in representation—namely, the use of the source 3D information—may also contribute, so it is hard to attribute it solely to a specific training strategy. The untextured evaluation also measures feature distances on rendered images and does not directly measure 3D shape accuracy. I understand the reasons for restricting the compared methods given the input/output conditions, but the concrete benefit in representative use cases should be made clearer.

Questions for authors
  1. Controlled experiment on the contribution of pretraining: Can you compare pretrained versus random initialization of the generator while keeping the same geometry-image representation, conditioning inputs, and VAE and generator architectures? I request a comparison that controls for differences other than initialization and states the set of trainable parameters as well as the training and hyperparameter-search budgets. Reporting learning curves and costs together would help distinguish final quality from training efficiency. If the shared pretrained VAE and DINO are kept, please state that the experiment evaluates the effect of generator pretraining. A comparison across training-data sizes is an additional suggestion that would strengthen the interpretation regarding data efficiency.

  2. Clarifying the target task and practical benefit: Which production task is this system most directly intended to improve? For example, if it is static expression-asset generation, please describe concretely the work required for input preprocessing/alignment and for output correction, and the advantages over existing approaches. Please also explain whether there are solutions to the difficulties expected when extending the method to video generation and similar settings, and what concrete applications can be put forward with the current static 3D faces and rendered images.

  3. Clarifying §3.3 and specifying reproduction settings: §3.3 describes the core guidance, but it took substantial effort to understand that z^t=zt−tvt\hat z_t=z_t-tv_t is the clean latent estimated at the current step, and to connect this with how the gradient of the expression loss is reflected in the next sampling step through the velocity correction. Please supplement this with a step-by-step description or pseudocode that specifies what is fixed and what is modified within a step, and through which modules gradients are backpropagated.

    In particular, for ∣∇vtLexp∣|\nabla_{v_t}\mathcal L_{\mathrm{exp}}| in Eq. 7, please clarify whether it denotes a vector norm or an element-wise absolute value; if it is a norm, which norm and over which axes it is computed; and the value of δ\delta. The two interpretations lead to different updates.

    Please also specify the number of inference sampling steps, the time schedule and solver used, the steps at which guidance is applied and the number of corrections per step, the LoRA rank and the layers to which it is applied, and the camera and lighting of the differentiable renderer as well as how the generated channels are used. Reporting the processing time and GPU memory usage with and without guidance, together with the inference hardware used, would make it possible to judge the computational cost. The training settings in Appendix A.2 are useful but do not sufficiently cover this inference and implementation information.

  4. Verifying the actual implementation of Eq. 8: The text states that cosine distance is used, but the printed equation applies a cosine once more to the normalized inner product. Letting s=⟨a,b⟩/(∥a∥∥b∥)s=\langle a,b\rangle/(\lVert a\rVert\lVert b\rVert), the usual distance is 1−s1-s, whereas the printed equation gives 1−cos⁡(s)1-\cos(s). The latter has a maximum of 1−cos⁡(1)≈0.45971-\cos(1)\approx0.4597 for s∈[−1,1]s\in[-1,1], which is inconsistent with the value 0.7979 in Table 2. Please confirm the formula used in the actual implementation and correct it if this is a typo. At present I cannot tell whether this is a notational error or an implementation difference, and I do not conclude from this inconsistency alone that the reported performance is incorrect.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.