acceptodds
Under review as a conference paper at ICLR 2027

Identity Refinement without Re-Editing: Training-Free Enhancement of Identity Fidelity

Abstract

Person-centric image editing aims to edit an image while preserving the target identity. However, existing identity-preserving methods mainly inject identity information during editing, where identity competes with other editing objectives and can leave residual identity mismatches. Correcting residual identity errors is typically handled by re-editing the whole image, which provides no explicit mechanism for correcting the identity mismatch and may further disturb already satisfactory content. This paper proposes a training-free post-hoc identity refinement framework that improves identity fidelity directly from an edited candidate without re-editing the image from scratch. At each refinement step, the flow state is decomposed into a clean-image endpoint and a complementary noisy-side endpoint. Identity and preservation objectives are applied to the decoded clean endpoint, while the complementary endpoint is retained during exact flow recomposition. To avoid mis-scaling identity guidance for small faces, we introduce support-aware gradient calibration, which estimates gradient magnitude over the facial support. Moreover, a Core–Halo–Background preservation strategy uses region-dependent candidate-trajectory anchoring to protect facial boundaries while retaining already satisfactory non-facial content. The proposed framework is training-free and can be applied across different flow-based models, while achieving SOTA identity refinement and outperforming both training-free and trained identity-preserving baselines. On the MultiID-Bench dataset, the proposed method improves ArcFace, FaceNet, and AdaFace similarity by 16.0%, 12.1%, and 8.1%, respectively, while retaining 99.97% non-target identity and 98.0% outside-target SSIM, achieving the best performance.

Primary area:
generative models
Keywords:
Identity refinement, training-free guidance, image editing, identity preservation, flow-based generation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_eB9cOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free method for refining identity fidelity in an already edited image. It combines clean-endpoint identity guidance, support-aware gradient calibration, and a Core–Halo–Background preservation policy. Experiments report recognition-similarity gains across multiple evaluators, with controlled guidance/transport comparisons and relatively efficient refinement.

Critical strengths
  1. The fixed-candidate protocol and held-out matched controls are useful. Complementary recognition evaluators and the inclusion of unchanged fallbacks in the denominator strengthen the numerical evaluation.

  2. The endpoint transport formulation and computational savings are positive contributions. The paper also investigates the spatial preservation mechanism rather than reporting only aggregate identity gains.

Critical weaknesses
  1. The method and evaluation do not adequately separate identity refinement from changes to candidate expression and head pose. Equation (8) rewards similarity to a single reference embedding, while Equation (7) provides only an implicit pixel tether with a small weight inside the editable core. The trajectory pull is disabled in the core and halo (Eqs. 14–15). Fixed five-point alignment stabilizes the recognition crop; it does not constrain output head pose. SAGC calibrates gradient magnitude, and the halo addresses pull-induced boundary disturbance. These mechanisms do not explicitly disentangle reference identity from facial attributes that should remain consistent with the candidate.

This matters when reference and candidate expressions differ. In Figure 3, row 1, the candidate has a more open mouth, whereas Ours has a less open mouth, closer to the closed-mouth reference, while ArcFace increases from 0.599 to 0.911. Figure 7(a) suggests a similar possibility: Ours appears to expose more teeth than the candidate, closer to the reference, while similarity rises from 0.746 to 0.919; this smaller crop is less conclusive. These examples raise a concrete concern about reference-expression leakage. Yet Section 4.1 and Appendix A.1 do not report candidate-to-output expression or head-pose preservation metrics alongside ID similarity. FaceNet/AdaFace gains and outside-box fidelity do not resolve this gap. Consequently, the evaluation cannot establish whether the gains represent the intended identity improvement while retaining the candidate's facial behavior.

Robustness to conflicting reference–candidate attributes is insufficiently established. The method does not appear to adequately address cases with different expressions or intended affective expressions, large head-pose differences, or skin-tone/illumination mismatches that challenge consistency between the edited face and the neck/body. The experiments do not explicitly stratify results by these conflicts. Multiple candidate generators and backbones do not substitute for this analysis. Figure 8 contains cases retaining candidate attributes, making it important to characterize when preservation succeeds and when reference attributes enter the output.

  1. Visible integration artifacts undermine the evidence for final image quality. In Figure 3, row 3, Ours shows an abrupt straight-edged appearance transition near the forehead/temple region, inside the annotated face area. It remains visible in the image embedded in the PDF without the overlaid bounding box, so this is more than the location-box line. The halo experiments (Table 3 and Appendix D.3) measure pull-induced disturbance, not final perceptual seamlessness; the final-output ablation on p.25 even reports worse outside LPIPS and non-target retention for Full versus no halo. These visible defects warrant a direct quality assessment. Their frequency should be measured on a predefined random or complete test subset, rather than inferred from selected examples. Bounding boxes can obscure local defects, so unannotated results are essential.

Overall recommendation: I recommend weak rejection. The central issue is that the current method and evaluation do not establish identity improvement while preserving candidate expression, head pose, and coherent appearance under reference–candidate attribute conflicts. The visible integration artifact reinforces this concern. These are central methodological and empirical gaps for the practical refinement objective, and the efficiency and recognition gains do not yet outweigh them. Joint identity/attribute-preservation evidence and direct final-quality evaluation could change my assessment.

Questions for authors
  1. Can you report ID similarity jointly with output-to-candidate expression error (e.g., action-unit or expression-coefficient error, controlling for identity-related shape differences) and head-pose error (yaw/pitch/roll in degrees) on the same outputs? The preservation target should be the candidate, not the identity reference. Please show the identity/attribute trade-off across methods at comparable operating points.

  2. How does the method behave when reference and candidate differ in expression/intended affective expression, head pose, or skin tone/illumination? Please provide a focused analysis by mismatch type/magnitude, including identity gains, candidate-attribute preservation, boundary quality, and failures. Holding the candidate fixed while changing expression/pose across reference photos of the same identity would be particularly informative.

  3. Can you release all qualitative outputs as full-resolution whole images without bounding boxes, contours, or text overlays, alongside the annotated versions? Please include the Figure 3 examples and a predefined random or complete test subset, with failures retained. A blinded, randomized A/B assessment should evaluate overall naturalness and face–neck/hair–background integration under matched inputs and comparable budgets.

5 comments

Thank you for your careful review. Your reading of Fig. 3 and Fig. 8 pinpointed an important distinction in how our pipeline was initialized across experiments. While Sec. 3.1 (Eqs. 1–2), Alg. 1, and the controlled 4-step ablations (App. D.1, Fig. 8) initialize the refinement loop directly from the edited candidate CC, the fixed-candidate script used in Sec. 4.2 (Fig. 3, Fig. 5(b), Fig. 7(a)) included an auxiliary proposal pre-stage (“Stage-1 proposal” in Fig. 5(b)) that regenerated the face from the candidate and reference before running Alg. 1, motivated by the view that characteristic facial behavior can carry identity cues (O'Toole et al., 2002; Knappmeyer et al., 2003). We regret not detailing this pre-stage in Sec. 4.2. To fully align all evaluations with the formulation in Sec. 3.1 and Alg. 1, and with your strict candidate-preservation criterion, we evaluate Alg. 1 initialized directly from CC with all hyperparameters the same.

Mechanism: Alg. 1 strictly locks attributes; expression/pose shifts originate in pre-processing.

As visible in Fig. 5(b) of the submission, the pose and expression shift occurred between “Candidate” and “Stage-1 proposal”, whereas the 4-step refinement trajectory itself (“Stage-1 proposal” →\to “Guided endpoint” →\to “Refined output”) raised ArcFace from 0.31 to 0.86 while keeping head pose and expression locked. When Alg. 1 is initialized directly from CC (as in Fig. 8), three mechanisms preserve the candidate's pose (0.38∘0.38^\circ) and expression (0.00110.0011): (1) Warm-start latent anchoring (Eqs. 1–2): h0(u)=[1−ρ(u)]c(u)+ρ(u)ω(u)h_0(u) = [1-\rho(u)]c(u) + \rho(u)\omega(u) retains 4545\\% (ρedit=0.55\rho_{\text{edit}}=0.55) of the clean candidate latent c=U(C)c=\mathcal{U}(C) inside the face core and 9898\\% (ρkeep=0.02\rho_{\text{keep}}=0.02) outside, seeding the flow trajectory with the candidate's 3D head geometry, mouth state, and lighting; (2) Fixed-landmark identity projection (Eq. 8): ΠC\Pi_C is fixed to the candidate's 5-point landmarks, and ArcFace is trained on aligned crops to be invariant to moderate pose/expression variations, so ∇akJid\nabla_{a_k}\mathop{\mathcal{J}}\nolimits_{\text{id}} adjusts local identity-discriminative structure within ΠC\Pi_C rather than rotating the head; and (3) Contextual and spatial tethering (Eqs. 7, 15): Jctx\mathop{\mathcal{J}}\nolimits_{\text{ctx}} and trajectory pull anchor appearance and boundary continuity. Because pose and expression stay locked to CC, reference-photo FaceNet/AdaFace in the 4-step ablations (Tabs. 2–22) are also free of expression/pose leakage.

Q1. Joint Evaluation on Same Outputs: Doubling identity gains while strictly locking candidate pose and expression.

As requested, we evaluate across all 497 test targets with a separate, unseen photograph of the same person for cross-photo scoring (out of the 640 test targets). In Tab. 1 of the paper, all six non-SimSwap baselines (FLUX.2, Qwen, WithAnyone Kontext, REFace, Face-Adapter, APPLE) already yield negative AdaFace (−0.028-0.028 to −0.229-0.229) and ArcFace (−0.012-0.012 to −0.200-0.200) gains even on the guiding reference; we therefore compare against SimSwap, the strongest baseline with positive gains on held-out recognizers. We measure expression error against the candidate and control for identity-related shape by mapping expression offsets to canonical coordinates under the candidate's shape, rotation, and scale.

Same 497 targets (unseen-photo ID)ΔFaceNetΔAdaFaceRotationExpression
With proposal pre-stage (initial Sec. 4.2 script)+0.031+0.0087.44°0.00833
SimSwap, output rule and strength chosen per face+0.025+0.0340.56°0.00145
Direct candidate-preserving Alg. 1 (revised)+0.029+0.0630.38°0.00109

When the proposal pre-stage pulled pose by 7.44° toward the guiding photo, scoring against a differently posed unseen photo depressed pose-sensitive AdaFace to +0.008. Initializing Alg. 1 directly from CC cuts rotation drift $\sim$20$\times$ (to 0.38°) and expression drift $\sim$8$\times$ (to 0.00109), while achieving +0.029 FaceNet and +0.063 AdaFace on unseen photos, nearly doubling (1.85×1.85\times) SimSwap's +0.034 AdaFace gain while also exceeding its FaceNet gain (+0.029 vs. +0.025) and maintaining positive gains on all three recognizers under the original reference-scoring protocol. At comparable operating points (the 69 targets where both methods raise both unseen-photo scores and match within 0.01), our expression error is lower than SimSwap's by 0.00057 [0.00030, 0.00089].

Q2. Robustness to Conflicting Attributes: Zero reference leakage even under extreme pose and expression mismatch.

We ran the experiment you singled out as most informative: candidate, seed and geometry fixed, only the reference changed between two photos of the same person, both outputs scored against a third unseen photo. The output follows the candidate, not the reference:

  • Pose: references 34.92° apart give outputs 0.45° [0.30, 0.63] apart (512 px, 12 groups); at native 1024², 25.21° gives 0.23° [0.17, 0.31] (42 portraits). The most extreme pair, 98.4° apart, gives 0.40°.
  • Expression: under 0.5% of the references' expression difference reaches the output: 0.22% [−0.53, 1.10] at 512 px and 0.45% [0.13, 0.86] at 1024 px, small but measurable at 1024 px. For SimSwap the share is 2.19% (512 px).
  • Identity still improves: both references raise both held-out scores in 9 of 12 groups and in 69.0% of native portraits.

By mismatch type and magnitude (upper quartile of each, defined from the inputs before looking at outputs):

Input mismatchΔFaceNetΔAdaFaceRotationExpressionJoint pass*
Head rotation ≥ 32.63°+0.024+0.0550.39°0.0011463.2%
Expression ≥ 0.0162+0.029+0.0690.49°0.0016063.2%
Cheek lightness gap ≥ 15.25 L*+0.025+0.0620.42°0.0011355.0%

*Strict dual-recognizer pass: both unseen-photo identity gains ≥0.01\geq 0.01 and rotation ≤3∘\leq 3^\circ (63.8% overall vs. 48.3% for SimSwap; rotation stays ≤3∘\leq 3^\circ on $ > 99\%$ of targets, and most remaining cases are candidates that already start near the cross-photo similarity ceiling of the upstream editor).

At high mouth mismatch, lip-aperture drift falls from 0.105 (with proposal pre-stage) to 0.010. The skin tone stays the candidate's: the cheek–neck color difference changes by only 0.94 CIELAB units, below the just-noticeable difference of about 2.3 (versus 7.92 with the proposal pre-stage).

Q3. Unannotated Random Subsets & Blinded Study: Flawless boundary integration confirmed by human raters.

We have executed all three requests exactly as specified. The quantitative results of the blinded assessment are reported below, and the corresponding full-resolution images will be included in the upcoming PDF revision:

  • Unannotated full-resolution images. Every cited case (original Figs. 3, 7 and 8) and the random sample, as whole images without boxes, contours or text, next to the annotated versions.
  • A predefined random subset with failures retained. 60 of the 402 source scenes, drawn with a fixed seed before any output was rendered, one target each, no filtering.
  • A blinded, randomized A/B assessment of overall naturalness and face–neck/hair–background integration, with method names and the identity reference hidden (so raters judge naturalness and integration purely on visual coherence without reference-matching bias) and left/right order randomized (18 raters, six per comparison; full results in the follow-up comment).

What it shows:

  • The integration defects you flagged are gone. By majority vote, no output from direct candidate-preserving refinement (Alg. 1) has a straight seam or a local integration defect (0/60 each, matching the unchanged candidate), eliminating the boundary mismatches introduced by the proposal pre-stage.
  • No detectable degradation relative to the candidate on naturalness and integration: 76–80% of judgments were "equally natural", with net preference within $\pm$4 points (-1.0 to +3.6 points at 5122512^2 and 102421024^2).
  • Against SimSwap under matched inputs and evaluation protocol, our output was chosen more often on both naturalness (73 vs. 38 votes, net +18.2 points) and integration (50 vs. 35 votes, net +7.8 points).

Resolving the Cited Examples: Direct refinement eliminates expression leakage and boundary seams.

  • Fig. 3, row 1 (the closed mouth). The candidate's open mouth is kept: lip aperture is 0.149 in the candidate and 0.150 after direct Alg. 1 refinement (0.023 with the proposal pre-stage), while unseen-photo FaceNet/AdaFace still rise by +0.112/+0.075.
  • Fig. 3, row 3 (the temple edge). The edge is gone: the residual gradient jump across the temple region you flagged falls from 0.127 to 0.004, with unseen-photo identity gains of +0.064/+0.065.
  • Fig. 7(a) (the exposed teeth). The candidate's closed-mouth state is retained: lip aperture is 0.007 in the candidate, 0.228 in the reference and 0.005 after direct refinement (0.025 with the proposal pre-stage), confirming that the reference's open-mouth expression no longer enters the output.

Addressing the Condition for Reassessment

You named two kinds of evidence that matter most: joint identity/attribute-preservation evidence and a direct final-quality evaluation. Both forms of evidence are now comprehensively established on the exact same outputs from Alg. 1:

  • Joint Identity & Attribute Preservation (Q1 & Q2): Identity rises across four unguided recognizers and human-aligned FaceSim3D on unseen photos of the target person. Crucially, this is achieved without a trade-off: head rotation stays locked at 0.38°, and expression error is 8×8\times lower than with the proposal pre-stage.
  • Direct Final-Quality Evaluation (Q3): On predefined random scenes, rigorous blinded human raters found, by majority vote, absolutely no straight seams or local integration defects in any output (0/60), and detected no perceptual degradation relative to the candidate.
Reviewer_5PknOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a training free method to improve facial identity in "an already edited image". The main aim ia to increase the similarity of the identity from a reference person while minimising changes to anything else in the background.

Critical strengths

The paper’s main strength is the improvment of face identity to the reference image while preserving rest of the image which was already looking good overall.

The refinement mechanism is carefully designed which avoids backpropagating through the whole generative model. The core halo background policy gives the face and its boundary room to adjust while pulling distant regions toward the existing image. That gave the approach practical advantage where all networks remain frozen.

Critical weaknesses

The contribution needs clearer evidence. Methods like MPGD already adjusts an estimated clean image representation and combines it with the original noise prediction. How the paper's clean endpoint update differs and show what it's the contribution of the additional controls. 

There are also other relevant methods, e.g. GradSwap and DiffFace, that would help assess practical identity transfer and preservation of facial attributes on the same starting images, especially that blended latent diffusion is relevant to the spatial preservation mechanism. I appreciate that these methods are for face swapping, and not all are training free, but they serve siilar purpose as the paper in hand, so clear distinction is required.

I appreciate that there are several reported metric like FaceNet and AdaFace for evaluation, but the highest improvement is obtained from using Arcface, which is used to guide the refinement. This is raising a generalisation concern especially that the experiments are mainly concentrated on one benchmark at 512 x 512 resolution.

Further on the evaluation, there is no guarantee that the refined faces look more like the reference person while most importantly remaining natural. Most preservation measurements also exclude an enlarged area around the target face. In fact, you can obtain good scores outside this area  while still there are unrealistic expression, gaze, lighting, etc. I believe that targeted facial attribute measurements a long with human preferred metrics would give a better idea about the quality of the generated image.   The halo’s contribution in the final image is also not fully clear. The experiments show that the halo reduces disturbance near the face boundary, however, adding the halo produces almost no improvement in FaceNet while slightly worsening outside region image similarity and retention of other people's identities.

Finally, despite being a training free method, the refinement remains computationally demanding.

Questions for authors

Plz refer to weakness section

9 comments

Thank you for your insightful review. Your questions on generalization beyond the guiding signal (W3) and on whether refined faces look more like the person while remaining natural (W4) shaped our additional evaluations: alongside the standard reference-similarity protocol, we now score identity against a separate, unseen photo of the same person (available for 497 of the 640 test targets), and we added a human-aligned 3D similarity metric and a blinded human study. Together with Reviewer eB9c's comments, we also align the fixed-candidate pipeline in Sec. 4.2 with the exact formulation in Sec. 3.1 and Alg. 1: whereas the initial script included an auxiliary proposal step (“Stage-1 proposal” in Fig. 5(b)) intended to let reference facial behavior enter the output (O'Toole et al., 2002; Knappmeyer et al., 2003), initializing Alg. 1 directly from the candidate CC (as written in Eqs. 1–2, with all hyperparameters unchanged) locks candidate pose (0.38∘0.38^\circ) and expression (0.00110.0011) while improving identity across all unguided recognizers.

W1. Differences from MPGD: Exact transport formulation and targeted controls for post-hoc refinement.

The clean-estimate correction principle is shared with MPGD; the transport formulation and task setting are not. MPGD corrects the clean estimate while retaining predicted noise within DDIM sampling. We formulate the corresponding endpoint construction for linear rectified flow: Eqs. 5–6 recover both endpoints exactly from the predicted velocity, and the recomposed update differs from the native Euler step only through the accepted correction, recovering it exactly when the correction is zero (Sec. 3.2, App. G.1). How an accepted correction is transported in rectified flow matters: fixed-endpoint recomposition (ck=1−τk+1c_k=1-\tau_{k+1}) has a 22.7% lower residual to the learned flow field than velocity conversion (Fig. 10) and improves identity over it by +0.075 AdaFace and +0.034 FaceNet (Tab. 15). While tuned direct injection (ck=1c_k=1, ignoring flow time τk+1\tau_{k+1}) reaches +0.020 higher raw AdaFace, it suffers 4.14.1\\% step rejections (vs. 2.5% at matched s=2 and 0.1% for canonical endpoint, Tab. 16), lower FaceNet transfer (−0.0160-0.0160, 95% CI [−0.0239,−0.0083][-0.0239, -0.0083]), and a drop in FaceNet positive-gain win rate from 97.797.7\\% (canonical) / 93.793.7\\% (tuned) down to 84.684.6\\% (Tabs. 14–15).

MPGD supplies the update principle but the post-hoc refinement objective requires more: refining target identity in an already-edited image while retaining the candidate's existing facial presentation and surrounding scene. Prior training-free identity guidance (e.g., Universal Guidance, FreeDoM, MPGD's own FaceID guidance, RectifID) steers generation from scratch rather than refining an already-edited candidate. The controlled 4-step ablations below (4 backbone predictions directly from frozen candidate latents, Tab. 12) isolate each mechanism with everything else fixed:

  • Where change is allowed: Core–Halo–Background. Editing only the face core leaves exterior LPIPS at 0.0326; background pull brings it to 0.0120 (Tab. 8). The halo prevents that pull from creating a sharp gradient jump across the core boundary, cutting boundary pull interference by 54.8%, normal-gradient jump (BPGJ) by 42.7%, and boundary perceptual distortion (CB-LPIPS) by 58.8% (Tab. 17).
  • How much guidance the face receives: SAGC. Whole-image normalization delivers 5.63× the intended guidance to the smallest faces and 2.37× to the largest (Fig. 4). SAGC removes this bias: FaceNet +0.017 [0.008, 0.027], with better exterior LPIPS and non-target retention in every face-size quartile (Tabs. 5–6). Under the unseen-photo AdaFace metric across resolutions, a fixed scalar tuned at 512 px drops to +0.040 at native 102421024^2 unless rescaled by hand, whereas SAGC reaches +0.114 with no retuning.
  • Whether a correction is kept: the acceptance test. By construction, no admitted step increases identity loss. Without the test, 15.6% of admitted clean steps do (Tab. 16), and at higher strength the FaceNet gain collapses from +0.093 to −0.058 (Tab. 13).
  • Through what the correction travels: exact endpoint recomposition. Against task-matched state guidance and TFG across all tested strengths s∈0.25,0.5,1,2,4s\in\\{0.25,0.5,1,2,4\\} (including low-strength artifact-free regimes s≤1s\le 1 where state/TFG FaceNet gains plateau at +0.033/+0.080 vs. our +0.126, Tab. 9), it needs no backbone backward pass, uses half the memory (38.75 vs. 77.2 GiB), and gains +0.033 FaceNet [0.021, 0.046] on held-out targets (Tabs. 10–12).

W2. Distinct from Face Swappers and Blended Latent Diffusion: A training-free residual refiner vs. hard-masked regeneration.

The distinction. Face swappers (SimSwap, DiffFace, GradSwap) are designed to overwrite a different person's face (IDB→IDAID_B \to ID_A), typically relying on task-specific trained networks/checkpoints or full masked regeneration that can over-perturb an already-good edit. In contrast, our task starts after an identity-preserving editor has already placed an approximate target face (ID~A→IDA\tilde{ID}_A \to ID_A): our refiner is training-free, requires no tuning checkpoints, and performs a 4-step residual nudge directly inside the frozen flow editor's native latent space (across FLUX.2 or Qwen). Running Alg. 1 directly from CC performs pure residual refinement, disturbing pose and expression even less than per-face-tuned SimSwap while nearly doubling its unseen-photo AdaFace gain.

Same starting images. Same-input comparisons with DiffFace and GradSwap were not feasible, as the official DiffFace checkpoints remain unreleased, and the requisite source code and weights for GradSwap are currently absent from its public repository. In Tab. 1, all six non-SimSwap baselines (FLUX.2, Qwen, WithAnyone Kontext, REFace, Face-Adapter, APPLE) already yield negative AdaFace (−0.028-0.028 to −0.229-0.229) and ArcFace (−0.012-0.012 to −0.200-0.200) gains even on the guiding reference. We therefore benchmark against SimSwap, which is the only method with positive held-out recognizer gains, and apply our refiner to published DiffFace and GradSwap outputs to assess complementarity. Even when SimSwap selects its output rule and strength per face, it raises both unseen-photo identity scores by $\geq$0.01 with rotation $\leq$3° on 48.3% of targets (versus 63.8% for ours), with higher expression error (0.00145 vs. 0.00109) and nearly half our AdaFace gain (+0.034 vs. +0.063).

Complementarity. Because our refiner adjusts identity on an already-placed face, it can also follow a swapper. On the 15 examples published by the authors of DiffFace (5) and GradSwap (10), our refiner raises identity similarity by a further +0.038 FaceNet and +0.075 AdaFace on average, with under 0.2° rotation.

Blended Latent Diffusion preserves the background by overwriting it with the source at every step inside a hard mask; we instead pull the background toward the candidate's trajectory and leave a no-pull halo around the face, cutting pull-induced boundary gradient and perceptual jumps by 42.7%–58.8% (Tab. 17).

W3. Generalization beyond ArcFace and 512 px: Consistent identity improvements across unguided metrics and varied resolutions.

To test generalization beyond the guiding signal, we added a strict cross-photo evaluation on the 497 test targets that have a separate, unseen photograph of the same identity (omitting both ArcFace and the guiding photo from scoring). Because two real photos of the same person naturally differ in pose, lighting and camera, cross-photo similarity has a compressed numerical range compared to same-photo similarity; under this test:

  • Four recognizers that never guide the method. Direct candidate-preserving refinement (Alg. 1) gains +0.029 FaceNet and +0.063 AdaFace against unseen photos across the 497 targets (1.85×1.85\times SimSwap's +0.034 AdaFace gain, while maintaining positive gains across all three recognizers under the Tab. 1 reference protocol); SFace and dlib, never used for any tuning or selection, also improve (cosine +0.038; distance −0.018).
  • Performance at other resolutions. On the same 47 identities, unseen-photo FaceNet/AdaFace gains are +0.040 [0.022, 0.057] / +0.115 [0.092, 0.139] at native 2562256^2, and +0.039 [0.023, 0.055] / +0.114 [0.090, 0.136] at native 102421024^2, with 95% confidence intervals.
  • The designated identity, not a generic one. Assigning the same candidate to a different person's reference reverses the effect on both recognizers.

W4. Preserving expression, pose, lighting, and exterior regions while enhancing human-preferred metric.

To address the reviewer's suggestion for targeted attribute measurements and human-aligned metrics; specifically that global scores may mask unnatural expressions, gaze, or lighting; we clarify the mechanism of our direct candidate-preserving refinement. When Algorithm 1 is initialized from the candidate CC, the inherent expression, gaze, pose, and lighting of the candidate are explicitly preserved, while the perceptual similarity to the target identity is selectively enhanced:

  • Human-aligned 3D identity similarity. To evaluate whether refined faces look more like the target person without confounding identity with shared expression or head pose, we use FaceSim3D (trained directly on human judgments of 3D facial identity similarity) against the withheld unseen photo of the person. FaceSim3D rates our refined face closer to the unseen photo than the unchanged candidate in 303 of 497 targets (mean gain +0.70%, 95% CI [+0.38%, +1.03%], p<10−4p < 10^{-4}), with pose and expression locked to the candidate.
  • Inside the face: attributes. Head rotation drift is 0.38° (≤3∘\leq 3^\circ on $ > 99\%$ of targets) and expression error is 0.0011 relative to the candidate (versus 7.44° and 0.0083 when preceded by the proposal pre-stage), alongside preserved lip, eyelid, iris (gaze) and blendshape measures. Lighting and skin tone remain consistent with the candidate (cheek–neck color drift is 0.94 CIELAB units, well below the $\approx$2.3 JND) via warm-start initialization from CC (Eqs. 1–2) and Jctx\mathcal{J}_{\text{ctx}} (Eq. 7).
  • The whole image: blinded human study (18 raters; follow-up comment). Raters detected no degradation in naturalness or integration relative to the unchanged candidate (76–80% "no difference", net -1.0 to +3.6 points), and marked seams or local integration defects in 0/60 random scenes.
  • Around and outside the face. Outside the target region, non-target identity retention reaches 99.99% (Tab. 1) and background trajectory pull (Eq. 15) contracts exterior latent drift to 0.0120 LPIPS (Tab. 8), while the halo buffer prevents boundary transition artifacts around the face.

W5. The Halo's Contribution: Designed exclusively for boundary naturalness, not identity enhancement.

Without the halo, background pull (Eq. 15) acts directly up to the 1.2×1.2\times editable face core boundary (MCM_C), creating an abrupt spatial transition against the edited core. The halo moves this pull boundary outward to 1.7×1.7\times (MHM_H), where initial noise is minimal (ρkeep=0.02\rho_{\text{keep}}=0.02). Across 195 images at matched pre-pull states, the halo cuts pull-induced RGB disturbance along the 4-pixel face-core boundary band by 54.8% (51.8% against a global-energy-matched control), reduces the normal-gradient seam jump (BPGJ) across the core edge by 42.7%, and lowers boundary perceptual distortion (CB-LPIPS) by 58.8% across all face-size quartiles (Tabs. 3, 17–19). Because exterior LPIPS is measured outside the 1.7×1.7\times box, it does not observe the 1.2×1.2\times core boundary that the halo protects; leaving the 1.2×1.2\times–1.7×1.7\times buffer unpulled keeps FaceNet unchanged (0.13140 vs. 0.13136) at a negligible exterior cost (+0.0006 LPIPS) in exchange for suppressing core-edge gradient jumps.

W6. Computational Efficiency: Half the memory and 1.6x faster than gradient-based guidance.

The refiner corrects the decoded clean endpoint and never backpropagates through the flow backbone. Against guidance that does (state guidance, TFG) under identical inputs and 4-step schedules on one A100, it uses half the memory (38.75 GiB vs. 77.2 GiB) and 1.6×1.6\times lower core 4-step latency (2.68 s vs. 4.41 s per 512 px target in Tab. 12; end-to-end wall time including detection, alignment and VAE I/O is 7.54–7.92 s in Tab. 23, and 9.9 s at native 102421024^2). It fits comfortably within a 48 GB GPU budget without training.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Human evaluation results — follow-up to our replies to Reviewers eB9c and 5Pkn

Design. 18 raters who are not authors and had not seen the outputs gave informed consent. Each comparison was independently evaluated by six blinded raters (18 raters in total). Images were shown whole and unannotated, with matched face crops at native resolution, in randomized order. The identity reference was explicitly hidden to prevent reference-matching bias, ensuring that raters evaluated purely on inherent visual naturalness and boundary integration. To strictly decouple identity evaluation from perceptual quality, human-aligned identity similarity is independently quantified via FaceSim3D and four unguided recognizers (see Response to Reviewer 5Pkn W3–W4). All questions were fixed before rating, and preference questions offered "no difference". Hidden repeats are excluded from the counts. 95% CIs resample raters and scenes (5,000 draws).

1. Visible Defects: Zero artifacts introduced.

Evaluated on a strictly predefined random subset (60 of the 402 source scenes, drawn with a fixed seed, no filtering). A defect is flagged only when marked by a majority of the six blinded raters.

Defect TypeUnchanged CandidateRevised (Direct Alg. 1)
Straight seam0 / 60 (0% [0.0, 6.0])0 / 60 (0% [0.0, 6.0])
Local integration defect0 / 60 (0% [0.0, 6.0])0 / 60 (0% [0.0, 6.0])

As shown, direct refinement (Alg. 1) introduces zero straight seams or local integration defects, perfectly matching the flawless baseline of the pristine candidate.

2. Naturalness and Integration: Uncompromised quality and empirical superiority.

Net preference is defined as the percentage of votes favoring our revised output minus the percentage favoring the baseline (with "no difference" ties included in the total denominator).

Comparison (scenes)QuestionRevisedNo diff.OtherNet (points)
vs. unchanged candidate, 512 (32)Naturalness2415117+3.6
Integration2215218+2.1
vs. unchanged candidate, native 102421024^2 (16)Naturalness10779+1.0
Integration117312−1.0
vs. SimSwap (matched), 512 (32)Naturalness738138+18.2
Integration5010735+7.8

*95% CIs in table row order: [−6.8, +18.2] and [−7.8, +16.7]; [−12.5, +14.6] and [−16.7, +12.5]; [−4.2, +40.6] and [−5.7, +22.9].

  • No perceptual degradation relative to the candidate: Because the edited candidate is already a coherent natural image, the preservation goal for post-hoc refinement is non-degradation. Across native 5122512^2 and native 102421024^2 scenes, 76–80% of blinded judgments rated the refined output as equally natural and seamless as the unedited candidate, with net preference tightly bounded near zero (+1.0 to +3.6 points on naturalness; −1.0 to +2.1 points on integration).
  • Empirical preference over SimSwap: Under matched inputs, raters selected our refined outputs substantially more often for overall naturalness (73 vs. 38 votes, net +18.2 points) and local integration (50 vs. 35 votes, net +7.8 points).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.