acceptodds
Under review as a conference paper at ICLR 2027

RetargetingFace: Bridging Heterogeneous Facial-Control Spaces with a Shared Expression Representation

Abstract

Facial animation systems encode the same expressions in incompatible native control spaces. Pairwise vector regression can bridge these spaces, but requires a separate mapping for each source–target direction. We therefore study cross-system facial retargeting as native-control interoperability: preserving the expression being conveyed while retaining each system’s original control parameterization. Based on this perspective, we introduce RetargetingFace, a shared expression interface that connects different facial-control systems using condition-level correspondence rather than exact cross-rig coefficient pairs. RetargetingFace extracts neutral-relative visual evidence from expressive and neutral renderings, reducing dependence on source facial shape while preserving expression-induced changes. Expression supervision, conditional rig confusion, and cross-rig alignment organize these observations into a shared representation. Each rig-specific predictor trains on its own latent–control pairs and is reused across source rigs. We construct a controlled GNM-derived dataset with 532 expression states, strength and FLAME-shape variation, and condition-matched native FLAME/ARKit realizations. On held-out FLAME↔ARKit transfers, RetargetingFace improves mean human similarity by 18.7% over the best evaluated existing baseline; the mean gain on external captured expressions is 9.9%. With the representation frozen, independently trained GNM and MHR predictors extend the interface to new target and source rigs. On the reported FLAME/GNM/MHR transfers, human ratings and first-hop cosine favor our method over direction-specific Direct MLP.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Facial Expression Retargeting, Cross-Model Retargeting, Visual Representation Learning, Cycle Consistency, Differentiable Rendering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_i7EVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes RetargetingFace, a framework for transferring facial expressions between systems with incompatible native controls. It learns a shared expression representation from expressive-minus-neutral DINOv3 features using expression supervision, conditional rig confusion, and cross-rig alignment. Rig-specific predictors are subsequently trained through same-rig coefficient reconstruction and reused across source systems. Validation includes learned expression-space similarity, coefficient-cycle recovery, and blinded human ratings.

Critical strengths
  • The authors evaluate additional native systems and expressions extracted from three sources and considered human evaluation

+The proposed Neutral-relative FE that is subtracting matched-neutral features aims to suppress static appearance information while retaining expression-induced changes, is a great idea and and sound. Experiments seems to confirm its relevance.

  • Authors consider human evaluation for this task which is crucial as often quantitative metrics used in the field do not reflect well human perception.

  • The controlled cross-rig dataset can help/enable systematic evaluation of expression transfer and I believe it can be a useful resource for 3d animation modelling/facial animation research.

Critical weaknesses
  • My main concern is with the paper’s writing. I had some difficulties reading it due to several reasons. The authors mention several acronyms/concepts without properly definig them since the abstract (like GNM). Overall I had difficulty following the manuscript, particularly the motivation and positioning of the contribution. The introduction explains the objective of reusing target predictors across source rigs, but it remains unclear what technical limitation of existing shared-representation approaches prevents this. What is the main methodological advance beyond combining a shared encoder with rig-specific predictors? This distinction needs to be developed more explicitly to assess the significance of the contribution. For instance , why we need a controlled cross-rig dataset is just discussed on section 3 (lines 116 to 120). However, even there the motivations/argument for this need is severely lacking.

  • Moreover, Related work section is largely presented as a sequence of brief descriptions rather than a connected argument establishing the unresolved problem and the proposed advance. Terms such as “shared expression interface” and “native-control interoperability” are repeated without making the methodological distinction substantially clearer. At the end I am still not sure if the paper has significant novel methodological contribution other than the controlled dataset.

  • Subsection "Expression coverage and native realization (lines 121 to 127) were quite confusing to me. Seems a mix of related/past work and proposal but the message is blurry. I would like the authors to clarify that in the rebuttal and manuscript. In addition, as the paper does not fully uses the 9 pages, it would be greatly appreciated if some sort of overview were presented at the beginning of section 3.

-The choice of DINOv3 as the visual backbone is insufficiently justified. Section 4.2 (lines 193–194) specifies its frozen CLS features, but the paper neither explains why these features are particularly suitable for capturing subtle facial actions nor compares them with other alternative backbones. So it is still unclear how much the results depend on DINOv3 and whether a simpler or face-specific encoder would suffice.

  • In my opinion, relative loss weighting of the training objectives is not sufficiently examined. The authors refer to Appendix A.5 for details, but this section combines many implementation choices, making the relevant information difficult to locate. It specifies equal weights for the terms in Equation (2), an alignment weight of 0.25, and a confusion weight increasing to 0.2, without explaining how these values were selected or assessing their sensitivity. Expression discrimination and rig invariance may impose competing constraints: for example, suppressing rig-dependent differences in how unilateral eye closure is realized could also remove useful expression details. A small sensitivity analysis would help establish whether the selected weights provide a robust balance between these objectives.

  • I also question whether the headline 18.7% improvement in human similarity is more like overselling the results (lines 24 - 28 and later lines 80 - 84) as it omits other comparisons /baselines like the Direct MLP). Appendix A.3 clarifies that this comparison excludes Direct MLP, the strongest evaluated baseline. Against it, the human scores are 2.33 versus 2.17 and 2.48 versus 2.32 (lines 299–305), approximately 7.1% average relative improvement. So the headline seems misleading.

  • The unseen-rig human results reported in Table 4 shows very small gain margins such as 2.13 versus 2.06 for FLAME→GNM, 2.11 versus 2.02 for MHR→GNM, and 2.14 versus 2.04 for MHR→FLAME.Are these gains consistent across source cases, or driven by a few examples? Since both methods are evaluated on the same 50 cases by the same 20 raters, paired confidence intervals could be computed from the existing ratings. In addition, I would like to see a paired analysis of the existing ratings for example, a paired t-test or a non-parametric case (Wilcoxon signed-rank test if differences is not normal), together with confidence intervals to assess whether these small differences provide reliable evidence of a perceptual advantage.

  • Lastly, to my understanding, the cross-rig supervision assumes that the same action and relative strength correspond to comparable expressions in both rigs. However, FLAME and ARKit coefficients are independently extracted and then scaled along their respective native paths. Does the same scaling produce comparable perceptual intensity? How frequently do the extractors disagree on the expression being conveyed? These points seems to impact the positive correspondences and as such should not be neglected.

Questions for authors

My main concerns are with the manuscript’s clarity and positioning, as well as the justification of several methodological choices and the strength of the empirical conclusions (see the critical weaknesses for details). At the current moment I lean on rejection mostly due to writing and issues in the experiments. Here are some more detailed points that I would like to see addressed based on the weaknesses raised in the previous section.

1)What is the main methodological advance beyond combining a shared expression encoder with rig-specific predictors? Can the authors clarify which technical limitations of the closest shared-representation approaches prevent the proposed functionality, and why a new controlled cross-rig dataset is necessary?

  1. How were the relative loss weights selected? Can the authors provide a small sensitivity analysis showing whether the conclusions remain consistent under alternative weights, particularly when balancing expression discrimination, rig confusion, and cross-rig alignment?

  2. Are the small human evaluation score gains (presented in Table 4) consistent across source cases? Can the authors provide paired confidence intervals and an appropriate paired statistical analysis (e.g., paired t-test or wilcoxon signed rank test) of the existing ratings whether these gains rare significant?

  3. What evidence supports treating the same action and relative strength as perceptually comparable across rigs? Since FLAME and ARKit coefficients are independently extracted and scaled, can the authors quantify disagreements in action or intensity on a representative sample and explain how these affect the positive correspondences used for training?

5)Is the 8.7% improvement the headline justified, given that it excludes Direct MLP from it (strongest baseline apparently)?

3 comments

Thank you for recognizing the value of the neutral-referenced features and the controlled dataset. Your questions helped us explain why the training design allows predictor reuse, how the experiments informed our choices, and what the reported gains mean.

Q1: What is the methodological advance beyond a shared encoder with rig-specific predictors, how does it differ from related work, and why is a controlled dataset needed?

A1: We learn expression correspondence across rigs, then train each predictor using only its own rig’s codes and controls. The ARKit predictor, for example, learns from (zA,cA)(z_A,c_A) pairs. Because the encoder has aligned the expression codes, the same predictor can take a FLAME code at inference and produce ARKit controls as DA(zF)D_A(z_F). We can therefore change the source and reuse the trained mapping.

For this to work, a code from the source rig must carry the expression information the predictor learned to use on the target rig. We reference expressive features to matched neutrals and use expression supervision to preserve differences in action and strength. Conditional rig confusion and multi-positive alignment encourage the two rigs to encode those expressions compatibly. We then freeze the encoder and train each native predictor separately. In Sec. 5.4, we test reuse by adding the GNM and MHR predictors with the representation and existing predictors fixed.

The closest methods differ in what their shared representations drive. Kim et al. (2021) synthesize a target image from a shared image latent and predict blendshapes from that image. SEREP (Josi et al., 2025) separates expression from identity and decodes a mesh. UniTalker (Fan et al., 2024) shares an audio-driven model across heterogeneous output annotations. In our method, an independently trained target predictor maps the shared visual code directly to that rig’s existing controls.

The controlled dataset supplies the two kinds of supervision these stages need. The encoder needs expressions matched across rigs, and each predictor needs the corresponding native controls. Captured images alone do not directly give us both. We therefore render source expressions with the Generative aNthropometric Model (GNM), an existing parametric head model, and extract native controls using SMIRK and MediaPipe. We link the rigs by matching action and strength. Rendering fixed coefficients on different FLAME shapes isolates shape variation, and rendering the same expression at successive strengths gives us an ordered sequence. The extracted coefficients then supervise each rig’s predictor. We organize the data by expression composition, source variant, strength trajectory, and shape, with held-out splits for evaluation.

As the revised abstract puts it (p. 1), “Each target predictor trains once on its own rig's expression codes and native controls, then decodes expression codes from different source rigs.” The Introduction defines the rigs and cites their sources. Sec. 2 discusses the closest methods, and Sec. 3 gives a worked example of the supervision.

Q2: How were the loss weights selected, and how sensitive are the results to the balance between expression discrimination, rig confusion, and cross-rig alignment?

A2: We selected a confusion cap of 0.20 and an alignment weight of 0.25 to bring the two rigs’ mean codes closer for the same action and strength while preserving retrieval of individual matches. We varied one weight at a time with the same seed and update budget. In all runs, the four expression terms had unit weight and 72 tuples each per update. We introduced rig confusion gradually after expression training began.

Table R3.1. Weight sensitivity at the same seed and training budget. The first numeric column in each panel gives the default. F = FLAME, A = ARKit. Strength order and mAP are percentages.

(a) Confusion cap (alignment weight fixed at 0.25)

Metric0.200.100.40
Rig-mean discrepancy ↓0.13350.15800.0933
F→A mAP ↑4.134.213.70
A→F mAP ↑3.243.302.94
Strength order ↑99.3399.2399.57

(b) Alignment weight (confusion cap fixed at 0.20)

Metric0.250.501.00
Rig-mean discrepancy ↓0.13350.14830.1592
F→A mAP ↑4.134.274.32
A→F mAP ↑3.243.413.53
Strength order ↑99.3398.9799.07

Raising the cap from 0.10 to 0.20 reduced rig discrepancy with little change in retrieval. The paired 95% intervals for the mAP change were [−0.18, +0.01] percentage points for F→A and [−0.13, +0.02] for A→F. At 0.40 the means moved closer still, but mAP fell by 0.43 [0.31, 0.54] and 0.30 [0.21, 0.40] points, respectively, relative to 0.20.

Increasing the alignment weight improved retrieval but widened the gap between rig means. We kept the default weight of 0.25, which also gave the lowest Validation rig separability (0.669, versus 0.723 at 0.50 and 0.686 at 1.00). Sec. 4.3 lists the training weights. Appendix A.5 (Table 13) reports the sensitivity results and paired intervals obtained by resampling the same 478 Test actions with checkpoints and galleries fixed.

Q3: Are the human-rating gains in Table 4 consistent across source cases, and what do paired confidence intervals and appropriate statistical tests show?

A3: In the supplied Table 4 scores, RetargetingFace had a higher mean rating on 27–39 of the 50 cases in each direction. We counted wins at the case level by averaging the 20 paired rating differences for each source case. Those same 50 case means enter the paired analysis, so every case carries equal weight.

Table R3.2. Case-level comparison of the supplied Table 4 scores. A win means that RetargetingFace had a higher case mean than Direct MLP.

DirectionMean gainAdjusted intervalWins / ties / lossesHolm p
FLAME→GNM+0.070[0.040, 0.105]27 / 23 / 02.98e-08
FLAME→MHR+0.340[0.239, 0.444]39 / 11 / 02.18e-11
GNM→FLAME+0.300[0.210, 0.393]37 / 13 / 07.28e-11
MHR→FLAME+0.100[0.059, 0.145]29 / 21 / 01.49e-08
GNM→MHR+0.120[0.070, 0.175]29 / 21 / 01.49e-08
MHR→GNM+0.090[0.053, 0.132]29 / 20 / 12.98e-08

We resampled cases using 200,000 paired bootstrap draws and applied a Bonferroni adjustment for approximately 95% simultaneous coverage across the Table 4 comparisons. All adjusted intervals lie above zero. Exact two-sided sign-flip tests using the same case-level differences gave a maximum Holm-adjusted p-value of 2.98×10⁻⁸.

The case-level analysis treats the observed rater panel as fixed. It assumes independent cases and, under the null, exchangeable method labels. To check sensitivity to the choice of raters, we also resampled cases and raters together. The adjusted intervals remained positive in every direction (Appendix A.7, paired analysis and statistical protocol).

Q4: What evidence shows that the same action and relative strength are perceptually comparable across rigs, how often do the independently extracted and scaled realizations disagree, and how do these disagreements affect the training positives?

A4: We form a cross-rig positive pair by matching action and relative strength, q=(k,s)q=(k,s). We tested how closely that label describes the rendered faces by comparing 9,560 renders per rig (478 actions × two held-out variants × ten strengths), with geometry fixed within each rig. We scored changes from neutral on both sets of renders with the same frozen MediaPipe readout. MediaPipe also supplies the ARKit labels.

Table R3.3. Agreement between extracted FLAME and ARKit expressions on strength and action, using a frozen MediaPipe readout.

ComparisonEstimate [95% action-bootstrap interval]
Strength vs. readout magnitude, FLAME (Spearman ρ)0.9730 [0.9665, 0.9787]
Strength vs. readout magnitude, ARKit (Spearman ρ)0.9873 [0.9855, 0.9890]
FLAME vs. ARKit strength ordering (Spearman ρ)0.9618 [0.9553, 0.9678]
Matched ARKit action closer than a different action to the FLAME query (%)82.71 [81.81, 83.60]

The first three rows use expression magnitude across all 51 channels. As we increased the strength of each action, the two rigs gave similar orderings (mean Spearman ρ = 0.9618). We then held variant and strength fixed and compared each FLAME expression with its matching ARKit action and all 477 alternatives. In this exploratory test, the match had higher cosine similarity on the 43 non-gaze channels in 82.71% of pairwise comparisons, counting ties as half.

To see where a positive pair can disagree, we looked at which eye closed more. In 1,517 pairs, the left and right blink scores differed by at least 0.05 on both rigs. The readout indicated that the same eye closed more on both rigs in 1,396 pairs (92.02%). In the other 121 (7.98%), it indicated opposite sides. Those pairs still share an action and strength label, so they can give the encoder a noisy signal about which eye closes more. During training, expression triplets and strength ordering constrain the codes alongside these cross-rig positives (Sec. 3, p. 3 and Appendix A.1).

Q5: How should the headline improvement be interpreted when Direct MLP, the strongest baseline, is excluded from that percentage?

A5: Direct MLP is the strongest human-rated baseline on the main transfers. On the 1–3 scale, our mean gains over it are +0.16 points in each direction (2.33 vs. 2.17 for FLAME→ARKit and 2.48 vs. 2.32 for ARKit→FLAME). The relative gain depends on which baseline supplies the denominator.

Table R3.4. Average relative gain in mean expression-similarity ratings for each choice of baseline.

Baseline used for relative gainMain transfersExternal expressions
Direct MLP7.1%8.4%
Strongest previously published method evaluated per setting18.7%9.9%

We compute the relative gain within each setting, then average the two main directions or the six external dataset–direction pairs equally (Appendix A.3). The main published baselines are MediaPipe for FLAME→ARKit and SMIRK for ARKit→FLAME. The revised abstract and Introduction report the Direct MLP gain first and identify the published baselines separately.

W4: The choice of DINOv3 as the visual backbone needs further justification.

Response to W4: We retained DINOv3 because it met our reconstruction criterion on both rigs. For each rig, we set the limit at 5% above the original same-seed predictor’s mean Validation NRMSE before examining the Test results. To compare backbones under that criterion, we retrained the Projector and both predictors with the same seed and training budget for each backbone.

Table R3.5. Change in mean within-rig Validation error relative to the original same-seed predictor. Lower values indicate better reconstruction.

BackboneFLAME change in mean NRMSEARKit change in mean NRMSERigs within the 5% limit
DINOv3+2.73%−1.32%2 / 2
DINOv2+6.93%−0.75%1 / 2
Reviewer_yB81OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RetargetingFace, a shared expression representation framework for facial expression retargeting across heterogeneous control systems, including FLAME and ARKit. It combines a DINOv3-based expression encoder with triplet supervision, rig confusion, and cross-rig semantic alignment, followed by rig-specific predictors that map the shared representation to native control parameters. Experiments on controlled datasets, external captured expressions, and human evaluations show improvements over direction-specific baselines, although generalization to unseen expression combinations and independently sourced new-rig expressions requires further clarification.

Critical strengths
  1. Modular reuse holds up consistently, including in human ratings. Predictors trained only on their own rig's data decode latents from other rigs. They beat a Direct MLP that uses paired cross-rig coefficients with the same architecture and budget in 14 of 14 human-rated settings: both base directions, six external dataset-direction settings, and six GNM/MHR directions. This consistency is the main evidence that the shared representation does real work, and it is what moves me toward acceptance.
  2. Careful experimental practice. Three seeds, a coefficient-locked readout test built after training, case-matched cosine-human agreement (including baseline-only pairs), and explicit statements of what each diagnostic does not show. These substantially mitigate, though do not remove, the concern that the main metric is the method's own embedding.
Critical weaknesses
  1. The primary metric is not independent of the method. S_emb is computed with the same frozen DINOv3 + projector the method uses at inference, so a predictor that inverts this encoder is favored by construction. In Table 2 only the proposed pipeline receives the matched neutral. Cosine-human agreement for baseline-only pairs in ARKit->FLAME is only moderate (68.9%; Spearman 0.44).
  2. Human-rated gains over the strongest baseline are small and untested. The margin over Direct MLP is +0.16 on a 1-3 scale in the base directions, 0.03-0.47 on external data and 0.07-0.34 for the new rigs. None of these margins come with significance tests or inter-rater reliability, and the new-rig margins have no intervals. The abstract's 18.7% / 9.9% figures exclude Direct MLP and so overstate the margin.
  3. The "unseen source rig" test is not independent of the model. GNM/MHR source renders are generated by the method's own predictors from FLAME latents. They therefore lie on the decodable manifold, which makes this closer to a cycle test than to genuine transfer. It is not stated how the Direct MLP pairs for these directions were obtained. FLAME->MHR cosine is only 0.54.
  4. Weak direct evidence that the latent is shared. Cross-rig exact-condition retrieval mAP is only 4.06% / 3.21% (Table 9).
  5. A narrow, synthetic setup. The data use one GNM source identity, one ARKit rig, and labels from two off-the-shelf extractors with linear strength scaling. Renders are untextured, frontal and gray. Held-out states only recombine the 19 seen base actions. The image-based baselines are run on these out-of-distribution renders, and there is no temporal evaluation.
Questions for authors
  1. Can you report at least one readout that does not use your projector for Fig. 3 and Table 4? Examples are a third-party AU/blendshape estimator, landmark-based geometric measures, or a projector with a different backbone or data split.
  2. Can you evaluate GNM/MHR as sources using expressions realized independently of your model (e.g., native GNM expressions, or MHR driven by its own controls)? How were the paired training data for the GNM/MHR Direct MLPs obtained?
  3. Please give paired case-level significance tests against Direct MLP, intervals for Table 4, inter-rater reliability, rater background, and the case-sampling procedure.
  4. Please report the headline gains relative to Direct MLP (about 7% in the base directions), or justify excluding it.
  5. How do a pivot Direct MLP (X->FLAME->Y) and per-rig predictors trained directly on 768-d DINOv3 neutral-difference features compare? The PCA reference (0.57/0.65) seems much weaker than I would expect from the latter.
  6. What are the gallery size and chance level behind the 4% cross-rig retrieval mAP, and how does it fit the claim of a shared interface?
  7. Were the training labels extracted from textured GNM images? Would textured renders change the ranking of the image-based baselines?
  8. How large is the disagreement between SMIRK and MediaPipe on the same expressions? Is invariance to the source face's shape tested at all for ARKit sources, given there is a single ARKit rig?
4 comments

We appreciate your close reading of the predictor-reuse experiments. Your questions about the source expressions, the baselines and the independent evaluation helped us explain how we tested reuse on new rigs and how we assessed the resulting transfers.

Q1: Can Figure 3 and Table 4 include a measure of expression preservation that is independent of the Expression Projector?

A1: For the FLAME↔ARKit transfers in Figure 3, we used MediaPipe to read 43 non-gaze blendshape channels from the source and retargeted faces. We subtracted each face’s neutral response and measured the mean absolute difference between the two expressions. Sec. 5.2 (p. 6) reports this readout for both methods on 9,560 matched Test cases per direction.

We also checked what happens to the decoded controls when only the source expression changes. With source identity and variant fixed, we decoded one expression that matched the reference action and strength and another that differed in action or strength. We then compared both outputs with the same stored target-rig coefficients.

Table R2.1. Percentage of comparisons in which the matching RetargetingFace output has lower mean absolute coefficient error. Ties receive half credit (chance 50%).

Alternative expressionFLAME→ARKit (%)ARKit→FLAME (%)
Different action, same strength94.3395.65
Same action, adjacent strength70.1471.40

Even at adjacent strengths, the matching output was closer more often than chance in both directions (Appendix A.10, Table 17, p. 35). These automated checks cover Figure 3. For the GNM and MHR transfers in Table 4, the independent assessment comes from the mean expression-similarity ratings reported alongside the Projector cosine.

Q2: How were the GNM and MHR source expressions obtained, and how were cross-rig training pairs constructed for Direct MLP?

A2: We obtain the expression controls for both source rigs before running RetargetingFace. GNM supplies its own native controls. To obtain the corresponding MHR expression, we extract ARKit blendshape weights from the GNM source image with MediaPipe and apply a fixed semantic mapping to the 72 MHR controls. Each source rig renders its controls at the designated strength. RetargetingFace encodes that render and passes the resulting code to the target predictor.

The same source image also links the two coefficient vectors used to train Direct MLP. For GNM→MHR, we pair the native GNM vector for a given expression, variant, and strength with the MHR vector derived from the same source image and scaled to the same strength. Direct MLP learns a separate mapping from such pairs for each ordered direction. The revised appendix walks through this example under Unseen-Rig Transfer Evaluation.

Q3: Who rated the expressions, how were cases selected, and what do paired tests, Table 4 intervals, and inter-rater reliability show?

A3: A dedicated team of 20 raters scored expression similarity on a three-point scale. Every rater saw every method on the same 50 source cases per direction, with method identities hidden and presentation order randomized.

For the main FLAME↔ARKit study, we selected 25 distinct actions for each of two held-out FLAME shapes from a fixed 1,000-case Test subset. This gave 50 cases, with ten at each relative strength from 0.6 to 1.0 in steps of 0.1. We kept each case’s action, source variant, and strength the same in both directions and used the same sampling procedure for Table 4.

On this scale, the reported mean gain over Direct MLP is 0.16 points in each main direction. For Table 4, it ranges from 0.07 to 0.34 points across directions.

Table R2.2. Paired score differences and ordinal agreement in the supplied Table 4 records. Gains are in points on the 1–3 scale.

ContrastMean difference95% intervalHolm-adjusted pOrdinal Krippendorff’s α
FLAME→GNM+0.070[0.040, 0.105]2.98 × 10⁻⁸0.849
FLAME→MHR+0.340[0.239, 0.444]2.18 × 10⁻¹¹0.859
GNM→FLAME+0.300[0.210, 0.393]7.28 × 10⁻¹¹0.844
MHR→FLAME+0.100[0.059, 0.145]1.49 × 10⁻⁸0.853
GNM→MHR+0.120[0.070, 0.175]1.49 × 10⁻⁸0.866
MHR→GNM+0.090[0.053, 0.132]2.98 × 10⁻⁸0.829

To estimate uncertainty in each gain, we first averaged the 20 paired rating differences within each case (RetargetingFace minus Direct MLP). We resampled the 50 cases 200,000 times per direction and used a Bonferroni adjustment for approximately 95% coverage across directions. We used the same case-level differences for the two-sided sign-flip tests and applied Holm adjustment across the six tests. For inter-rater agreement, each case–method combination contributed one item to ordinal Krippendorff’s α, giving 100 items per direction with 20 scores each.

Q4: What is the gain over Direct MLP, and which methods are used to calculate the headline percentages?

A4: RetargetingFace gains 0.16 points in each main direction over Direct MLP, the strongest human-rated baseline on those transfers. Table R2.3 gives mean ratings on the 1–3 scale and the absolute gain for each comparison.

Table R2.3. Mean expression-similarity ratings for Direct MLP and RetargetingFace. Gains are absolute differences on the 1–3 scale.

Evaluation setFLAME→ARKit (MLP → Ours)GainARKit→FLAME (MLP → Ours)Gain
Main Test2.17 → 2.33+0.162.32 → 2.48+0.16
KDEF-dyn2.10 → 2.18+0.082.21 → 2.40+0.19
FaceGrabber2.01 → 2.04+0.032.05 → 2.52+0.47
Multiface1.95 → 2.12+0.172.22 → 2.33+0.11

For Direct MLP, averaging the relative gains equally across directions or dataset–direction pairs gives 7.1% on the main transfers and 8.4% on external expressions. The 18.7% and 9.9% figures use the strongest previously published method evaluated in each setting. On the main transfers, that baseline is MediaPipe for FLAME→ARKit and SMIRK for ARKit→FLAME. The revised abstract gives the Direct MLP comparison first, and Sec. 5.2 and Appendix A.3 identify the baselines and the averaging rule.

Q5: How well do rig-specific predictors perform on the full 768-dimensional DINOv3 feature difference, and how well do two Direct MLPs perform when composed through FLAME?

A5: We standardized the full 768-dimensional DINOv3 expression-minus-neutral feature on Train and trained each predictor on its own rig. With the same seed and 9,560 matched Test cases per direction, the learned Projector more than halved cycle NRMSE relative to the full feature in both directions (Sec. 5.3, Table 3(b), p. 8).

Table R2.4. First-hop cosine and cycle error for predictors trained on the 768-dimensional feature difference or the learned expression code.

TransferCosine ↑ · 768-D differenceCosine ↑ · learned ProjectorCycle NRMSE ↓ · 768-D differenceCycle NRMSE ↓ · learned Projector
FLAME→ARKit0.53260.88680.91610.4501
ARKit→FLAME0.57960.95840.83250.3570

The second baseline uses FLAME as an intermediate rig. Its first frozen Direct MLP predicts 55 FLAME coefficients, which the second frozen MLP maps to the target. On the same 9,560 cases per direction as Table 4, we scored the final render against the source using cosine similarity. We then re-encoded that render and measured how closely the cycle recovered the original source controls using cycle NRMSE (Sec. 5.4, p. 8 and Appendix A.7, Table 15, p. 30).

Table R2.5. Direct transfer and transfer via FLAME between GNM and MHR, evaluated on the same 9,560 cases per direction.

MethodGNM→MHR cosine ↑GNM→MHR cycle NRMSE ↓MHR→GNM cosine ↑MHR→GNM cycle NRMSE ↓
Direct MLP0.34010.93750.32430.8295
Two Direct MLPs via FLAME0.33850.93760.32460.8386
RetargetingFace0.60050.82470.67070.8469

Routing through FLAME changed the MLP’s endpoint cosine by less than 0.002 in either direction and produced slightly higher cycle NRMSE than single-hop regression in both. Compared with the relay, RetargetingFace had higher cosine in both directions and lower cycle error on GNM→MHR. The relay had lower cycle error on MHR→GNM (Table R2.5).

Q6: How large is the retrieval gallery, what mAP would random ranking achieve, and what does the observed mAP tell us about the shared interface?

A6: To count as a match, a candidate from the other rig must have the same action and strength as the query. Queries and gallery candidates come from disjoint source variants. The gallery contains five variants per condition, and each FLAME variant appears on two shapes. This gives the match counts in Table R2.6. Average precision measures how early those matches appear in the ranking.

Table R2.6. Cross-rig retrieval of the same action and strength. The last three columns report mAP for random ranking and for encoders trained without or with alignment.

TransferGallery sizeMatches per queryRandom mAP (%)Without alignment (%)With alignment (%)
FLAME→ARKit23,90050.06131.974.06
ARKit→FLAME47,800100.04261.293.21

On identical queries and galleries, second-stage alignment more than doubled mAP in each direction (Appendix A.4, Table 12 in the revision, Table 9 in the original submission). With alignment, expressions that share an action and strength rank earlier across rigs.

Q7: Were the training labels extracted from textured GNM images, and can input texture change the ranking of image-based baselines?

A7: Yes. SMIRK and MediaPipe extracted the native controls from GNM images rendered with detailed skin texture. The gray faces shown in the main benchmark come one step later, when we render FLAME and ARKit faces using those controls. Section 3 now distinguishes these two steps.

To test whether texture affected the ranking, we rendered the same 19 peak expressions on three GNM identities, once with uniform skin color and once with detailed skin texture. Within each pair, we averaged the two images’ 51 MediaPipe blendshape scores channel by channel to obtain a common reference. We then rendered each method’s output on the same gray FLAME face, applied MediaPipe, and measured the mean absolute error against that reference.

Table R2.7. Effect of source texture on baseline rankings. Values are mean absolute errors against the common MediaPipe reference across 57 matched render pairs.

Image baselineUniform skin color ↓Detailed skin texture ↓
EMOCA v20.05480.0526
EMICA0.06340.0623
SMIRK0.06710.0470

Against the common reference, SMIRK ranked third on uniformly colored inputs and first on textured inputs. Its rank also improved under each of the other reference choices when we averaged errors across the same 57 pairs. Appendix A.1 gives the rendering protocol and results for all three references.

Q8: How closely do the expressions extracted by SMIRK and MediaPipe agree, and did you test sensitivity to face shape when ARKit was the source?

A8: We compared SMIRK-driven FLAME and MediaPipe-driven ARKit renders across 9,560 matched conditions. For both rigs we used the same 51-channel MediaPipe readout and subtracted the neutral response (Sec. 3, p. 3 and Appendix A.1).

Table R2.8. Agreement between FLAME and ARKit renders on expression magnitude and eye closure, measured from changes relative to neutral.

Agreement between rigsResult
Mean absolute difference per channel0.0349
Ordering of expression strength (mean Spearman ρ)0.962
Agreement on which eye closed more1,396 / 1,517 (92.02%)

For the eye comparison, we included pairs where the left–right blink difference was at least 0.05 in magnitude on both rigs.

We tested sensitivity to face shape with FLAME (Sec. 5.2, Table 2(a)), where we could vary the identity shape parameters while holding the expression coefficients fixed. Our ARKit rig has 51 expression blendshapes on a single neutral mesh and no separate identity shape parameters, so we used the same source face shape for all ARKit transfers.

Dear Reviewer, As we are nearing the end of the discussion period, we hope you have had a chance to look at our rebuttal and the revised manuscript. If they address your concerns, we would appreciate it if you could consider updating your rating; if not, please let us know what remains. Thank you for your time and feedback.

Reviewer_g6uuOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a method to perform facial expression transfer across different face mesh representations (that are unaligned) by learning an a shared expression latent space. During inference, this expression space can be utilized by a strength and action conditioned decoder to transfer the source facial expression from one representation (e.g FLAME) to another target expression (e.g ARKit). This is done via a careful curation of data on which each component of the model is trained. The expression encoder is trained via a triplet loss that pushes the latent to match the target expression and be invariant to identity (using both FLAME and ARkit spaces). Quantitative results show good retargeting performance though it uses its own latent space for measurement and not ground-truth data.

Critical strengths
  1. The paper tackles an important problem of expression transfer across different face mesh representations.

  2. Proposed method makes sense, I can see why it would work

Critical weaknesses
  1. The paper is written poorly and hard to follow. The story is laid out in extremely broad strokes in the main paper but the crucial details to connect the story to a concrete methodology lies in the Appendix. Significant chunks of the appendix belong in the main paper to clarify detail and disambiguate the writing. Poor writing and presentation are the main drawbacks of this paper.

  2. Most quantitative results (except for those using human judgement) use the latent space of the method itself to evaluate baselines and the proposed method, this can be misleading if the latent space is not meaningful. Thus, there should be more comparison with controlled ground-truth data by using the same identity and different expressions (this could be be treated as another unseen ‘Rig’ system)

Questions for authors

N/A

3 comments

Thank you for suggesting a comparison that holds identity fixed while varying expression. We used that comparison to test whether a predictor trained on one rig preserves differences in action and strength when it decodes expressions from another. The revised text explains how the predictor is trained and how we tested its outputs.

W1: The main text needs a clearer account of how the shared encoder and rig-specific predictors are trained and used at inference.

Response to W1: The Expression Projector learns which expressions correspond across rigs, and each predictor learns how its own rig realizes them. A worked example in Sec. 3 (p. 3, “Expression Coverage and Native Realization”) shows where both training signals come from. “Action k=1k=1 combines blowing with lowering the mouth corners.” For variant 0, SMIRK and MediaPipe extract the FLAME and ARKit peak coefficients from the same image. Scaling each rig’s coefficients by 0.6 before rendering gives us two examples of the same action at relative strength 0.6, each expressed in its own native controls.

With the visual backbone frozen, we train the Projector on FLAME and ARKit features matched by action and strength. We then freeze the Projector and use its codes to train each predictor on its own rig’s native coefficients. Sec. 4.4 (p. 5, Eq. 6) shows how the predictor produces native controls c^r=Dr(zr)\hat{\mathbf c}_r=D_r(\mathbf z_r) and how the reference controls cr\mathbf c_r supervise its training outputs Pr(zr)P_r(\mathbf z_r).

At inference, the ARKit predictor that learned from (zA,cA)(\mathbf z_A,\mathbf c_A) can take a FLAME expression code and return c^A=DA(zF)\hat{\mathbf c}_A=D_A(\mathbf z_F). It can serve different supported sources because their expression codes share this representation. All modules stay frozen, and inference requires no action or strength labels. We clarified the training and inference steps in blue in Sec. 3 (p. 3), Sec. 4.1 (p. 3), and Sec. 4.4 (p. 5).

W2: Evaluation in the learned latent space needs a controlled comparison with identity held fixed and expression varied.

Response to W2: We tested whether the encoder recognizes an expression across face shapes and whether the decoder preserves it when identity is held fixed.

For the first test, we rendered the same 72 action–strength conditions on different FLAME shapes and gave every encoder a matched neutral (Table 2(a)). R@1 measures how often the top match on another face shape has the same action and strength.

Table R1.1. Exact action–strength retrieval across FLAME shapes with a matched neutral for each encoder.

EncoderMatched cosine ↑Match–nonmatch gap ↑AUC ↑R@1 (%) ↑
EMOCA v20.91290.30710.965453.01
EMICA0.94500.27850.975454.21
SMIRK0.83950.37280.939342.92
RetargetingFace0.96180.76750.999596.48

Our encoder retrieved the exact condition in 96.48% of queries, compared with 54.21% for EMICA, the strongest baseline on R@1. We next held identity fixed and tested whether those expression differences survived decoding into the target rig’s native controls.

For each of the 9,560 Test cases per direction (478 actions × two held-out variants × ten strengths), we decoded two expressions from the same source identity and variant using the same frozen encoder and target predictor. We compared both outputs with the stored target-native coefficients for the first expression. Table R1.2 reports how often its matching output was closer to that reference, using mean absolute error across 51 ARKit or 50 FLAME expression channels.

Table R1.2. Fixed-identity decoding. Entries give the percentage of comparisons favoring the matching output, with 95% action-bootstrap intervals.

Different-condition outputFLAME→ARKit (%)ARKit→FLAME (%)
Same action, adjacent strength70.14 [69.48, 70.78]71.40 [70.63, 72.17]
Same action, any other strength88.28 [87.75, 88.79]90.93 [90.50, 91.35]
Other action, same strength94.33 [93.96, 94.69]95.65 [95.24, 96.05]

We included every case, averaging first within each query and then equally over variants, strengths and actions. Counting ties as half gives a chance level of 50%. Even when the two inputs differed by just one strength level, the matching output was closer to the reference more often than chance in both directions (Sec. 5.2, Table 2(b), p. 6 and Appendix A.10, Table 17, p. 35).

Dear Reviewer, As we are nearing the end of the discussion period, we hope you have had a chance to look at our rebuttal and the revised manuscript. If they address your concerns, we would appreciate it if you could consider updating your rating; if not, please let us know what remains. Thank you for your time and feedback.

Thank you for the detailed rebuttal and the additional experiments. I appreciate the authors' efforts to address my concerns.

However, some limitations remain. First, the evaluation of new rigs still relies on expressions constructed through a common source and semantic mapping pipeline, leaving generalization to independently sourced expressions insufficiently established. Second, the independent evaluation is only partial, since MediaPipe also participates in constructing the ARKit supervision. Third, although the cross-rig retrieval results improve substantially over chance, the relatively low absolute retrieval performance and limited diversity of the evaluation setup leave some uncertainty about the robustness of the shared representation.

Overall, I consider the rebuttal constructive and the paper's empirical evidence considerably stronger. Nevertheless, I do not think the remaining limitations justify upgrading the paper to a stronger acceptance recommendation. I therefore maintain my rating of 3.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted detailed responses to your reviews.

Please review their feedback, engage with the authors if any points remain unclear, and update your evaluation and score if their response has addressed your concerns.

Thank you

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.