acceptodds
Under review as a conference paper at ICLR 2027

MetaHuman-Anything: Vision-Language Face Similarity Grounding for Engine-Friendly Digital Human Sculpting

Abstract

Automated creation of editable 3D facial assets is increasingly important for digital human production in games and animation. However, image-based 3D face reconstruction methods typically output point clouds, meshes, or parametric face models rather than native characters that can be edited directly in animation engines. MetaHuman's mesh-conforming tools fit a native face template to reconstructed meshes, a step that can alter facial geometry and reduce identity fidelity. We present MetaHuman-Anything, an automated, vision–language-model-guided framework that directly edits native MetaHuman parameters to match a reference portrait. At each iteration, a frozen vision–language Reasoner identifies differences between the reference and the current render. A state-conditioned Grounder translates this analysis into grouped parameter targets and gated updates, while a learned renderer provides feedback for iterative refinement. We construct a corpus pairing rendered faces with editable MetaHuman states to train the framework. On an external set of 102 portraits, our method improves mean target-to-Unreal-Engine-render ArcFace identity similarity by 118.2% over the strongest mesh-conversion baseline under matched fixed-appearance conditions, while producing directly editable native MetaHuman characters.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
engine-native digital humans, MetaHuman, vision-language models, 3D face reconstruction, facial modeling, iterative refinement, character generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7ivgOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents MetaHuman-Anything, an automated, vision–language-model-guided framework that generates editable MetaHuman characters from reference portraits by predicting updates directly in the native parameter space. It uses visual comparisons and surrogate-rendering feedback for iterative refinement and constructs a native face-state corpus containing 2,000 identities and 15,000 state–render pairs. Experiments show that the method preserves more facial identity in final Unreal Engine assets than the evaluated reconstruction-then-conversion baselines.

Critical strengths
  1. The method addresses the practical need for editable native MetaHuman assets. The method directly edits MetaHuman parameters, and the native face-state corpus provides paired supervision in the same parameter space. It addresses a concrete need for creating characters that can be edited in the target engine.

  2. MetaHuman-Anything improves identity similarity in the final Unreal Engine assets. Under matched fixed-appearance conditions, it achieves higher mean UE ID-Sim than the evaluated reconstruction-then-conversion baselines on the external London portraits.

Critical weaknesses
  1. The method is designed specifically for MetaHuman, but the paper provides limited background and importance of MetaHuman and why its native assets are useful for character creation. Maybe the MetaHuman Character/DNA and the choice of the initial native state need more explanation to make the workflow clear. Figure 1 is not explicitly referenced or discussed in the introduction.

  2. The reasons for choosing 3DDFA-V2, DECA, and MICA, and the settings used to convert their outputs to MetaHuman, are not adequately discussed.

  3. The different appearance settings in Figure 3 make facial identity harder to compare visually. The baselines use fixed appearance, while MetaHuman-Anything is shown with author-configured skin and hair, as disclosed in Sec. 5.1. Adding fixed-appearance renders of MetaHuman-Anything would allow a direct visual comparison under the same conditions as Table 1.

  4. A central motivation of the paper is that converting reconstructed meshes to MetaHuman may reduce identity fidelity. However, Table 1 only compares the final UE results of different complete pipelines, without reporting identity or geometric errors for the same reconstruction before and after conversion. The results therefore do not show how much of the observed difference comes from reconstruction quality, conversion loss, or supervision in the native parameter space.

Questions for authors
  1. What are the feature definitions, model architecture, and training procedure for the lightweight mapping from GNM identity features to predicted DNA features, and what rules are used to select identities after conversion (Sec. 3.2)?

  2. What are the complete input formats for the Reasoner and Grounder, including any text instructions used and the placement of <REG> and <REASON_REG> in the Grounder input (Sec. 3.3) ?

Reviewer_xhPNOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper maps reference portraits to editable native MetaHuman characters. A frozen vision-language Reasoner compares the target with the current preview, while a state-conditioned Grounder predicts grouped parameter targets and update gates. A learned renderer provides differentiable previews for training and iterative inference; final characters are saved and evaluated through Unreal Engine. Training uses a corpus of 15,000 rendered states spanning 2,000 identities, supplemented by identity supervision from real photographs. Experiments report improved rendered identity similarity over reconstruction-and-conversion baselines and analyze reasoning inputs, iteration, and identity conditioning.

Critical strengths
  • Useful native output with final-engine validation. Predicting native controls yields editable characters and avoids reconstruction-to-MetaHuman conversion at inference. On 102 London portraits under common fixed appearance and rendering conditions, the selected editor obtains UE ID-Sim of 0.1631 versus 0.0747 for MICA, improving 91 portraits. Evaluating actual saved characters, rather than only surrogate previews, makes this evidence consequential for the intended application.

  • Substantial implementation and informative diagnostic reporting. The paired corpus uses identity-disjoint splits, and the paper evaluates surrogate fidelity and several input interventions. The authors explicitly acknowledge that the benefit of pairing reasoning with its own portrait remains unresolved. This transparency helps delimit what the experiments support.

Critical weaknesses
  • The central reasoning and refinement mechanisms are not convincingly isolated. Fresh reasoning achieves 0.16307, but shuffled reasoning achieves 0.16155; the paired difference is only 0.00153 with a 95% interval crossing zero. This does not establish equivalence, but leaves the claimed value of portrait-specific visual reasoning unsupported. Removing reasoning gives 0.11514, yet this inference intervention changes the input distribution and does not replace a model trained without reasoning. Appendix B.3 also reports different batch sizes for fresh and shuffled conditions. Furthermore, the one-to-eight-round gain is small (0.00145) and measured on the initial editor, not the selected model. The evidence therefore remains insufficient to attribute the main improvement to matched visual reasoning and iterative correction.

  • The strongest comparison does not separate mechanism from training and output representation. The main London comparison demonstrates an advantage over reconstruction followed by conversion, but does not isolate the proposed mechanism from direct native prediction, identity embeddings, and real-photo identity supervision. Static Grounder and one-shot baselines are included in internal validation; however, these do not provide a matched final-engine comparison on London. The selected editor also changes FFHQ sampling and identity-loss weight from 0.10/0.05 to 0.50/0.50, alongside replay training. A static predictor with comparable inputs, objectives, data, and training budget is needed to establish the additional benefit. MetaHuman specialization itself is reasonable, but the current results leave the general methodological contribution uncertain.

Questions for authors
  • Mechanism attribution. Can fresh and shuffled reasoning be evaluated with identical batch sizes, and one versus eight rounds be compared on the selected checkpoint? Is there a matched model trained without reasoning, or with generic reasoning, that can distinguish dependence on this input from dependence on its portrait-specific content?

  • Matched native prediction baseline. Can the static Grounder using identity input and the 0.50/0.50 settings in Table 5 be evaluated on London through final UE rendering? Please clarify remaining training-budget and objective differences. This comparison could materially change my assessment of the iterative method. Testing another backend would be useful future work, rather than a prerequisite for this submission.

Reviewer_wjHSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a new method for digital human generation from a single portrait. The introduced framework is built on MetaHuman, where two VLM-based models sharing the same backbone, i.e., the Reasoner and the Grounder, are adopted to predict the parameters and iteratively refine the generation results by comparing the reference image and the current render. For collecting the training data, this paper introduces a GNM-to-MetaHuman pipeline for supervised learning and conducts self-supervised learning on FFHQ. Experimental results show that the proposed method surpasses the previous methods including MICA, DECA and 3DDFA-V2.

Critical strengths
  1. The proposed framework makes sense: digital human generation is a complex task, and using a VLM to provide commonsense knowledge and conducting task-specific training is beneficial for digital human generation.

  2. Introducing MetaHuman as the base parametric humanoid model is also reasonable: this model offers greater degrees of freedom and generalization capability, and its editability makes the technology more practical.

Critical weaknesses
  1. Data construction is somewhat limited. The GNM-to-MetaHuman pipeline leads to relatively low data quality: GNM itself has limited degrees of freedom, and the conversion process introduces further losses. Additional real 3D face data should be supplemented to improve data quality.

  2. The experimental comparison is not fair enough. In the quantitative experiment (Table 1), the ID-Sim of the original reconstructions before MetaHuman conversion should be included in addition to UE ID-sim. In the qualitative comparison (Figure 3), hair should be removed before comparison, since a suitable hairstyle can greatly enhance perceived similarity. In addition, more visualization results are necessary; currently only four comparison groups are shown in Figure 3.

Questions for authors
  1. Hope the authors will discuss whether using 3D scans of real faces is meaningful.

  2. Could the authors provide the ID-Sim scores for the original reconstructions before MetaHuman conversion?

  3. Could the authors provide larger-scale visualization results, especially showing cases both with and without hair?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.