acceptodds
Under review as a conference paper at ICLR 2027

Mind the RefGAP: Correcting Reference Attention in Diffusion-Based Visual Editing

Abstract

Reference-guided diffusion editors struggle to faithfully reproduce user-provided references. We identify a potential bottleneck in diffusion editors: many methods provide limited reference-attention allocation. For example, in LoomVideo, edit-region queries assign less than 1% of their attention mass to the reference. We introduce RefGAP, a training-free correction that determines logit-offset magnitudes online at each layer from the reference-attention mass measured during the forward pass. Positive offsets to reference logits strengthen reference usage by edit-region queries, while negative offsets for keep-region queries limit reference-induced changes outside the edit. Two global coefficients control the correction; they are selected once on validation data from four development diffusion editors and held fixed. Across seven diffusion-based image/video editors, RefGAP improves identity fidelity in head swapping and face swapping. RefGAP achieves a fidelity-preservation trade-off comparable to separately tuned constant edit-side biases, without per-approach strength sweeps. Additional experiments on virtual try-on and background replacement evaluate transfer beyond identity editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reference-guided image and video editing, diffusion models, attention analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1sstOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2(+1)
Presentation
3(+1)
Soundness
3(+2)
Read review
Paper summary

RefGAP is a training-free method that uses regional reference-attention statistics to strengthen reference conditioning inside edited regions and suppress it elsewhere. Evaluation across seven image and video editors demonstrates consistent identity-fidelity improvements, with preservation trade-offs and mixed results on other tasks.

Critical strengths
  1. The paper provides useful empirical insights into reference-attention behavior across different editors. It examines attention allocation across layers and between edited and preserved regions. These observations offer a useful diagnostic perspective on how editing models use reference information.

  2. The method is simple and demonstrates practical value across multiple editors. It requires no additional training or test-time optimization. Consistent identity improvements across seven editors, including three excluded from coefficient calibration, support its transferability.

Critical weaknesses
  1. The novelty over existing attention-control methods is insufficiently established. AnyScene [1], BooW-VTON [2], and KV-Edit [3] provide related spatial-control precedents, while SaaS [4] already uses regional attention statistics for adaptive reweighting. The paper should clarify the distinct contribution of its particular reference-attention correction.

  2. The analysis does not adequately justify the proposed mass-dependent correction. The attention profiles and parameter ablations do not establish that reference-attention mass reliably predicts editing failures or the required correction strength. In particular, the keep-region ablations demonstrate the benefit of suppression without establishing the benefit of its adaptive form.

  3. The computational cost analysis does not fully substantiate the claimed tuning savings. Although Table 6 reports inference overhead, the costs of offline statistical analysis, initial calibration, and profiling each new model remain unquantified. These development and setup costs should be reported separately and included in comparisons with simpler alternatives.

  4. The writing and presentation need improvement, particularly the clarity of Figure 1. Its unbalanced layout and small annotations make the proposed attention intervention difficult to follow.

References

[1] AnyScene: Customized Image Synthesis with Composited Foreground. [2] BooW-VTON: Boosting In-the-Wild Virtual Try-On via Mask-Free Pseudo Data Training. [3] KV-Edit: Training-Free Image Editing for Precise Background Preservation. [4] Scale Your Instructions: Enhance the Instruction-Following Fidelity of Unified Image Generation Model by Self-Adaptive Attention Scaling.

Questions for authors

Could the authors further explain the novelty of the proposed method and provide more evidence of its effectiveness, addressing the concerns raised in the weaknesses? I also suggest improving the writing and presentation, as this would help readers better understand the paper.

9 comments
AuthorsOpenReview ↗

Novelty over the cited works.

Thank you for the careful review, and we are sorry the paper was hard to follow. To restate the idea in one sentence: RefGAP measures how much attention the edited region pays to the reference image, and adds an offset sized by that measurement, raising reference attention inside the edit and lowering it outside.

W1. Novelty over the cited works. Thank you for pointing out these prior works; the revised Related Work section (Sec. 2) now cites and discusses all four. We believe, however, that RefGAP is a new design that differs significantly from them: none of them both raises attention to the conditioning tokens in the edit region and lowers it in the keep region, with a strength set from measured attention. AnyScene and BooW-VTON learn region control through training; at inference, AnyScene also zeroes foreground-word cross-attention outside a fixed mask, a hard cut rather than a graded reduction, with no edit-side gain. KV-Edit copies the source's keys and values into the background. SaaS is the closest, but it rescales attention only inside the edit region:

KV-Edit SaaS RefGAP
Strength set from– Matching input-image attention Reference attention only
RequirementOutput tokens aligned with the input Input-image tokens in the same attention as the reference; undefined on VACE, LoomVideo Reference tokens only (all seven editors)

A natural question is whether SaaS would work just as well if it were applied to reference tokens instead of text. We tried this: inside our edit mask, we rescaled reference attention with SaaS's rule, choosing its scale ξ = 2 with the same procedure we use for RefGAP. We also ran KV-Edit. The table reports ID ↑ / background LPIPS (LPIPS_ne ↓) on the head-swap test set; ✗ marks a background damaged by more than .005 relative to the base:

Klein Klein-base Qwen-Image-Edit OmniGen2
base.726 / .090 .588 / .158 .609 / .289 .207 / .262
SaaS rule.740 / .090 .602 / .160 .658 / .290 .286 / .264
RefGAP, edit side only.783 / .100 ✗ .776 / .293 ✗ .802 / .338 ✗ –
RefGAP.790 / .071 .799 / .087 .723 / .075 .693 / .192
KV-Edit background lock:   
KV-Edit.752 / .036 .632 / .042 .426 / .041 .218 / .052
KV-Edit + RefGAP edit side.791 / .040 .799 / .061 .776 / .045 .706 / .059

Even with a tuned scale, SaaS's rule raises ID by at most .079 on its five editors (JoyAI in Table 30), while RefGAP raises it by +.06 to +.49. The table also shows why both sides are needed: RefGAP's edit side alone damages the background, and the keep side prevents this without extra forward passes. KV-Edit's hard lock can replace the keep side and preserves the background even better, but it doubles the forward passes, needs token-aligned output, forbids any change outside the mask, and raises Kpt on Qwen-Image-Edit (.213 vs .179; Table 30).

We hope this makes the distinction clearer, and we are happy to discuss any of these comparisons further.

AuthorsOpenReview ↗

Mass-dependent correction

W2. Mass-dependent correction. Thank you; this comment helped us sharpen the claim. We use the measured mass to set the correction strength, not to predict how much an editor will gain or which samples will fail, and Sec. 3.2 now says so explicitly.

Does the mass track the strength an editor needs? We find that it does (App. H.5, Figure 6). RefGAP's online offset ranges from 1.48 on Klein and Qwen-Image-Edit to 3.21 on LoomVideo, which has the lowest mass and is also the editor that needs the largest tuned constant (2.5). How the offset varies across layers and steps matters as well: a constant equal to LoomVideo's mean offset breaks preservation (ΔLPIPS_ne +.018 vs +.003).

Why not a fixed strength? A simple baseline would remove the dynamic part of our framework and use the best fixed strength instead, chosen with the same validation procedure as RefGAP. We tested it, and it cannot adapt to every case:

  • the best fixed strength depends on the editor (b from 1.5 to 2.5 when tuned per editor), so every new editor needs its own search;
  • one strength shared across editors is too weak on LoomVideo, the lowest-mass editor (ID −.018);
  • on the four image editors, the selected shared strength damages the background more (details below).

This is why RefGAP sets the strength from the measured mass. Table 4 compares it with the best fixed edit-side strength chosen per editor (highest ID on validation with ΔLPIPS_ne ≤ .005; only the edit side differs). Without that search, RefGAP comes within .006 in ID (head-swap test set, whole-frame LPIPS):

EditorRefGAP ID / Kpt / LPIPS Tuned b Constant ID / Kpt / LPIPS
JoyAI.644 / .085 / .215 1.5 .644 / .075 / .207
LoomVideo.665 / .098 / .275 2.5 .671 / .098 / .271
Klein.790 / .153 / .217 2.0 .794 / .171 / .229
Klein-base.799 / .236 / .261 2.0 .801 / .236 / .260

For the shared case, we selected one pair of fixed offsets with RefGAP's rule on the development editors (+1.5 edit, −1 keep) and ran it on the head-swap test set. Here we also count near-copies: outputs whose whole-frame LPIPS to the reference is below 0.2, where the editor reproduces the reference frame instead of editing the input (Figure 11 shows examples from the constant). Because copying inflates ID, one column scores near-copies as 0. Results as RefGAP / constant:

EditorNear-copy % ↓ ID, includes near-copy ↑ ID, near-copy = 0 ↑ LPIPS_ne ↓
Klein7.0 / 9.4 .790 / .788 .730 / .708 .071 / .083
Klein-base6.4 / 7.0 .799 / .795 .744 / .736 .087 / .093
JoyAI1.5 / 1.7 .644 / .644 .631 / .631 .079 / .072
LoomVideo7.5 / 6.2 .665 / .647 .609 / .598 .136 / .133
VACE2.3 / 2.3 .729 / .736 .709 / .716 .047 / .047
Qwen-Image-Edit9.1 / 22.3 .723 / .738 .645 / .560 .075 / .129
OmniGen25.3 / 10.3 .693 / .745 .650 / .663 (n.s.) .192 / .202

Bold: better arm (difference ≥ .005 or 1 point, 95% CI excluding 0). The near-copy difference on Klein-base is borderline, and the constant's higher raw ID on Qwen-Image-Edit and OmniGen2 partly comes from copying; on VACE it is .008 higher (App. H.1, Tables 24–26).

On the keep side, suppression itself is needed: the keep-side ablations (Table 5) show that lowering reference attention in the keep region helps, and the edit side alone damages the background (table in W1). Its mass-dependent form is a design choice (Sec. 3.3); we do not claim it outperforms a well-chosen fixed keep offset, and App. C.1 now says so. The evidence for mass-dependent strength is on the edit side, above.

AuthorsOpenReview ↗

W3. Setup cost.

W3. Setup cost. Thank you for asking for this; App. A now reports the setup cost. Every option below needs some parameter selection. The difference is that RefGAP does it once, while a per-editor constant has to repeat it for each new editor:

RefGAP Shared constant pair Per-editor constant
Parameter selectionOnce, on 4 editors (18 configs, 32 GPU hours), then fixed Once, on 4 editors (5 × 5 grid of (b, c), 44 GPU hours), then fixed Repeated on every editor (6 values, Table 4)
Adding a new editor0.02–1.8 GPU hours (one profiling run) 0.02–1.8 GPU hours (one profiling run) 0.1–11 GPU hours (a new sweep)

GPU hours are estimated from the per-clip times in Table 6.

Reviewer_WW3COpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RefGAP measures the attention mass assigned to reference tokens separately for edit and keep queries, then derives positive and negative reference-logit offsets from these measurements. Two shared coefficients and a layer-selection rule are calibrated on four development editors; each new editor still requires validation profiling. Experiments cover seven image/video editors, head and face swapping, virtual try-on, and background replacement. Identity fidelity improves across all seven editors, with dependence-aware uncertainty estimates and human evaluations. The paper also documents pose/quality trade-offs, a background-replacement failure, and implementation costs.

Critical strengths
  1. Broad and statistically careful evidence for the main benefit. Identity fidelity improves across seven editors, and the paired uncertainty analysis accounts for shared identities and reference content rather than treating frames or clips as independent (Table 2; Appendix E). This makes the practical identity-editing benefit convincing.

  2. Experiments reveal both the usefulness and boundaries of the intervention. The paper evaluates calibration, keep-side suppression, layer coverage, tuned constant edit-side biases, and human preferences, while explicitly disclosing failures and costs. This systematic evidence substantially improves understanding of reference conditioning and supports acceptance despite the intervention's simplicity.

Critical weaknesses
  1. The necessity of mass-dependent correction is not fully established. The constant-bias comparison changes only the edit-side rule, retaining adaptive keep-side correction and the proposed layer sets, and is reported on the development editors. It does not exclude a simpler shared pair of constant edit/keep offsets transferring similarly. Appendix C.1 also acknowledges the missing comparison for adaptive keep-side suppression. This uncertainty limits how much of the benefit can be attributed specifically to online mass dependence.

  2. Identity gains can incur consequential preservation and quality costs. Head-swap keypoint error increases from .043 to .085 on JoyAI and from .097 to .153 on Klein (Table 2). In the unscreened audit, all 22 non-tied visual-quality judgments favor the base, although they are concentrated in five items and most judgments are ties (Appendix F). Delayed correction reduces geometry costs while also reducing identity gains. These results support an identity-prioritizing operating point, but do not establish an improved overall trade-off at matched preservation.

Questions for authors
  1. Bounded validation of the adaptive-design claim: Can one shared pair of constant edit/keep offsets be selected on the development editors and compared with RefGAP on the existing held-out editors, retaining the same layer-selection rule? This would test whether online mass dependence is necessary for the reported transfer.

  2. Clarification and analysis of existing results: What identity-fidelity advantage remains when the available strength/delay configurations are compared at similar pose-preservation levels? If the intended contribution is specifically identity-prioritizing editing, please make that scope explicit. The answer could change my assessment of the practical trade-off.

5 comments
AuthorsOpenReview ↗

W1 / Q1. Does one shared constant pair transfer as well?

Thank you for the precise and constructive review. Your two suggested experiments go to the heart of what RefGAP claims, so we ran both exactly as specified.

W1 / Q1. Does one shared constant pair transfer as well? To isolate the role of online strength, we kept RefGAP's layer rule and keep-side suppression and replaced only the strength with fixed offsets: +b on the edit side and −c on the keep side. We selected (b, c) = (1.5, 1) on the 4 development editors with RefGAP's own rule (App. C; App. H.1, Tables 22–23). Because both sides become constant, this tests their online form jointly, not the keep side alone.

One measurement issue turned out to matter. If an editor copies the reference image into the output, ID goes up, and LPIPS_ne cannot tell this apart from other background damage. We therefore also count near-copies, i.e., outputs whose whole-frame LPIPS to the reference is below 0.2 (Figure 11 shows the constant's near-copies), and report ID with them scored 0. Results on the head-swap test set (1,040 clips), as base / RefGAP / constant:

EditorNear-copy % ↓ ID, includes near-copy ↑ ID, near-copy = 0 ↑ LPIPS_ne ↓ Kpt ↓
Klein7.9 / 7.0 / 9.4 .726 / .790 / .788 .673 / .730 / .708 .090 / .071 / .083 .097 / .153 / .152
Klein-base10.9 / 6.4 / 7.0 .588 / .799 / .795 .534 / .744 / .736 .158 / .087 / .093 .169 / .236 / .234
JoyAI1.7 / 1.5 / 1.7 .507 / .644 / .644 .495 / .631 / .631 .075 / .079 / .072 .043 / .085 / .074
LoomVideo5.6 / 7.5 / 6.2 .608 / .665 / .647 .566 / .609 / .598 .133 / .136 / .133 .107 / .098 / .100
VACE (held out)2.1 / 2.3 / 2.3 .405 / .729 / .736 .390 / .709 / .716 .048 / .047 / .047 .198 / .200 / .195
Qwen-Image-Edit (held out)57.7 / 9.1 / 22.3 .609 / .723 / .738 .153 / .645 / .560 .289 / .075 / .129 .179 / .179 / .194
OmniGen2 (held out)5.6 / 5.3 / 10.3 .207 / .693 / .745 .188 / .650 / .663 (n.s.) .262 / .192 / .202 .255 / .258 / .249

Bold: better of RefGAP and constant (difference ≥ .005 or ≥ 1 point, 95% CI excluding 0; Kpt by size only). CIs in App. H.1 (Tables 24–25).

Our reading of these results:

  • On the four image editors, RefGAP makes fewer near-copies (borderline on Klein-base) and preserves the background better.
  • On LoomVideo, the shared constant is too weak (ID −.018). Its b is capped at 1.5 because b ≥ 2 breaks preservation on JoyAI, while RefGAP chooses a larger strength there online.
  • In the plain ID column (before scoring near-copies as 0), the possibility you raised holds: the shared constant also transfers and is higher on all three held-out editors. On Qwen-Image-Edit and OmniGen2, however, part of that lead comes from copying. On Qwen-Image-Edit the constant makes 13.2 points more near-copies, and RefGAP is .084 higher once they score 0; on OmniGen2 the constant doubles the near-copies, and its lead of .052 shrinks to .028, .013, and .000 at thresholds 0.15, 0.2, and 0.25 (Table 26). On VACE, the constant stays .008 higher.

So, without choosing b per editor, what online strength buys is mainly less copying and better preservation on the image editors, and enough strength on LoomVideo. This is consistent with what the paper claims, a trade-off comparable to tuned constants without per-editor strength sweeps, and the comparison is now in App. H.1. Thank you for suggesting this test.

AuthorsOpenReview ↗

W2 / Q2. ID at similar pose preservation.

W2 / Q2. ID at similar pose preservation. A clear identity gain remains. If we pick, on validation, the highest-ID setting whose Kpt stays within .005 of the base, RefGAP still raises ID on all seven editors (+.03 to +.49). Four editors meet this at the default; the other three need a weaker or later correction (Table 20):

EditorRefGAP setting Base ID / Kpt RefGAP ID / Kpt ΔID
Qwen-Image-EditDefault .609 / .179 .723 / .179 +.114
VACEDefault .405 / .198 .729 / .200 +.324
LoomVideoDefault .608 / .107 .665 / .098 +.057
OmniGen2Default .207 / .255 .693 / .258 +.486
Kleinγ_edit 0.4, skip 1 .726 / .097 .758 / .104 +.032
Klein-baseBias only for σ ≤ 0.9 .588 / .169 .695 / .170 +.107
JoyAISkip step 1 .507 / .043 .613 / .052 +.106

When we sweep strength and delay for RefGAP, and b (at c = 1) for the constant, their ID–Kpt curves largely overlap (App. G.2, Figure 4).

Scope. Thank you for this suggestion; it helped us state the contribution more precisely. RefGAP is indeed meant as identity-prioritizing reference editing, and the revised abstract now says so. Its first priority is to keep reference content out of the non-edit region (W1); within that, it favors identity over exact pose, which is why the default corrects at all steps (Sec. 5, App. G.2). The balance remains adjustable: when keypoint consistency matters more, starting the correction later (or lowering γ_edit) brings Kpt back near the base while keeping a clear ID gain (+.03 to +.11 on the three editors where Kpt rises; table above).

We hope this answers your question about the practical trade-off, and we would welcome any further thoughts.

AuthorsOpenReview ↗

W2. Image quality.

W2. Image quality. RefGAP aims to improve reference fidelity while leaving image quality largely unchanged, and the data support this in most cases. In the unscreened audit, 92.1% of quality judgments are ties; the 22 decided ones, which favor the base, come from just five items rather than 22 distinct ones (App. F). As a no-reference check, MUSIQ (0 to 100) changes by at most 2.5 points on 11 of 14 head- and face-swap pairs. The three larger drops, LoomVideo (head −4.0, face −8.2) and Klein-base face swap (−5.1), are listed in Limitations.

Reviewer_AmPaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on a very narrow gap in diffusion editors where some details within the reference image are not copied correctly. The paper proposes a training-free editing adjustment technique that adjusts how much the model uses the reference image, depending on which part of the output it is working on.

Critical strengths
  • Human evaluation across four tasks
  • Good performance across the paper’s benchmarks and other editors; Tables 2 and 4; in Table 4, it performs about as well as carefully adjusted settings without needing that adjustment for each editor (Table 4).
Critical weaknesses
  • Results are strongest for head and face swapping, while background replacement is mixed and becomes substantially worse
  • The paper tests several editors but limited tasks and situations. It remains unclear whether their proposed approach also helps with replacing everyday objects or editing videos with complex movement and multiple people.
Questions for authors
  • Table 2 provides good evidence that RefGAP improves existing editors; Kpt errors also increase in several settings and also LPIPS worsening in some settings, how do you explain its increase?
  • How does your method perform on other benchmarks for object replacement and challenging video editing?
6 comments
AuthorsOpenReview ↗

W1. Background replacement.

Thank you for the careful reading and for noting that Table 2 shows RefGAP improves existing editors. Your suggestions to test more tasks and to explain the Kpt and LPIPS changes led us to two new experiments and a clearer analysis; we reply to each point below.

W1. Background replacement. Thank you for flagging this. We acknowledge that RefGAP fails on background replacement with Qwen-Image-Edit (background DINO .299 → .042), and we record this as a limitation in Sec. 5. Among the 28 editor–task pairs in Table 2, this is the only one where reference fidelity drops; on the other six editors, background DINO rises or does not change significantly (App. E). App. G.3 analyzes the case: the reference here is a scene rather than a person, so pulling the edit toward the reference can conflict with keeping the person intact, and a fixed edit-side bias fails in the same way. The failure is therefore not specific to RefGAP's adaptive strength.

AuthorsOpenReview ↗

W2 / Q2. Everyday objects and multiple people.

W2 / Q2. Everyday objects and multiple people. Thank you for this suggestion; it was a good test of how general RefGAP is. We ran both settings with the parameters used in the paper, without any retuning, and have added them to the revision (App. H.2–H.3, Tables 27–29; examples in Figures 9–10).

Object replacement. We used DreamEditBench for the four image editors (212 pairs) and DreamSwapV for the three video editors (167 tasks). Subject DINO measures how close the new object is to the reference, and background LPIPS measures how much the area outside the object changes (lower is better). The table reports RefGAP − base with 95% CIs:

EditorΔBg. LPIPS ↓ Subject DINO ↑ ΔDINO
Klein−.052 [−.069, −.035]  .603 → .626  +.023 [+.006, +.042]
Klein-base−.239 [−.310, −.169] .596 → .639 +.043 [+.011, +.076]
Qwen-Image-Edit−.175 [−.238, −.116] .626 → .634 +.008 [−.018, +.035]
OmniGen2−.045 [−.073, −.019] .583 → .627 +.044 [+.020, +.070]
JoyAI+.004 [+.002, +.005] .163 → .217 +.054 [+.039, +.070]
LoomVideo+.006 [+.004, +.008] .150 → .164 +.014 [+.004, +.025]
VACE+.002 [+.000, +.003] .281 → .305 +.025 [+.004, +.045]

With RefGAP, the object moves closer to the reference on six of the seven editors; the exception, Qwen-Image-Edit, is not significant. On the image editors the background is also better preserved, and on the video editors it changes by at most .006.

Multiple people. We cropped 97 clips with several visible faces from 92 VFHQ videos and swapped one head per clip; both base and RefGAP are told the target's position in the prompt. Target ID compares the swapped person with the reference (scored 0 if no face is found there, which keeps absolute values low), and non-target ID checks whether everyone else still looks like themselves:

EditorTarget ID ↑ ΔID ΔNon-target ID ↑
Klein.522 → .597  +.075 [+.058, +.094]  +.021 
Klein-base.640 → .707 +.067 [+.020, +.117] +.031
Qwen-Image-Edit.121 → .173 +.052 [−.000, +.104] −.026 [−.055, −.004]
OmniGen2.091 → .374 +.282 [+.221, +.344] +.088
JoyAI.034 → .092 +.058 [+.041, +.076] −.004
LoomVideo.138 → .193 +.055 [+.032, +.080] −.002
VACE.083 → .232 +.150 [+.111, +.191] −.002

Target ID rises on all seven editors (the CI excludes 0 on six), and the other people keep their identity. The small drop on Qwen-Image-Edit comes from a few clips (6 of 95 fall by more than .1) in which the editor redraws a bystander instead of the target. Outside the target head, LPIPS_ne rises by at most .0022 (+.0052 on JoyAI).

Taken together, these results confirm that RefGAP applies beyond head and face swapping: with the same parameters, it also improves object replacement and multi-person head swapping, which further supports the generality of the method. Thank you again for pointing us to these settings.

AuthorsOpenReview ↗

Why do Kpt and LPIPS increase?

Q1. Why do Kpt and LPIPS increase? Thank you for this question. In short, RefGAP leans toward identity: when the editor attends more to the reference, it takes in mostly the identity, but also a little of the reference's pose and expression. We found two mechanisms behind the Kpt increase:

  • Head swap: on the three editors where Kpt rises (JoyAI, Klein, Klein-base), the outputs move toward the reference's expression (.06 to .26) and head pose (0.2° to 3.7°), beyond what an unrelated reference produces (App. G.2, Table 19).
  • Face swap: here Kpt is measured against the input face, and facial landmarks depend on face shape, so part of the rise is a measurement effect: a successful identity change raises Kpt by itself. On Qwen-Image-Edit, LoomVideo, and OmniGen2, the rise is concentrated in the clips with the largest ID gains. On JoyAI and Klein-base, however, Kpt still rises when ID changes little, so a real pose and expression cost remains there, as in head swap (App. G.1, Table 18).

Human raters, comparing pose/expression consistency with the input, mostly judge ties (Table 3). Among the non-tied judgments (RefGAP vs base), RefGAP is better on head swap (27 vs 3), where whole-frame LPIPS also does not increase on any editor (Table 2). On face swap the base is better (19 vs 40), consistent with the remaining cost noted above.

When pose matters more, the balance can be shifted: γ_edit scales the edit-side offset, and the correction can also start later. If we pick, on validation, the highest-ID setting whose Kpt stays within .005 of the base, RefGAP still raises ID on all seven editors (+.03 to +.49). Four editors already meet this at the default; the settings for the other three are:

Editor   Setting   Base ID / Kpt   RefGAP ID / Kpt
Klein   γ_edit 0.4, skip step 1   .726 / .097   .758 / .104
Klein-base   Bias only for σ ≤ 0.9   .588 / .169   .695 / .170
JoyAI   Skip step 1   .507 / .043   .613 / .052

Figure 4 shows the full ID–Kpt curve on four editors (App. G.2, Table 20).

On the other tasks, the LPIPS increase is limited: it rises by .005 or more on 7 of 21 editor–task pairs (at most +.036), and each of these also gains reference fidelity (+.035 to +.309). Part of the increase simply reflects that the base often changes little (e.g., JoyAI face swap, Figure 7).

We hope this explains where the cost comes from and how it can be controlled; we are happy to discuss further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.