acceptodds
Withdrawn from ICLR 2027

Reprise: Identity-Faithful Human Video Generation from Image and Video References

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Subject-driven video generation has made substantial progress in synthesizing videos of a referenced person in novel scenes, yet existing methods typically condition on a single reference image and measure identity by facial similarity between the generated face and that same reference. A reference captures the person under one specific viewpoint, styling, lighting, and expression, and is thus a sample of an identity rather than the identity itself, so same-reference protocols reward copying such transient attributes rather than preserving the person. The issue becomes more severe in multi-person scenarios, where no token-level cue binds each subject to its own references, leading to subject confusion and attribute swapping. We propose Reprise, a multi-reference identity-preserving video generation framework in which each character is specified by multiple image and video references from different contexts, and identity is defined as what remains invariant across these contexts. We construct Reprise-10K, a corpus of 10K identities derived from film and television, in which each identity is paired with multiple reference images and video clips spanning distinct viewpoints, lighting, expressions, and styling. We further introduce ID-RoPE, which extends rotary position embedding with an additional identity coordinate carried by every token, grouping attention by character so that all references of a character are attended jointly rather than any single one being favored, and avoiding the copying shortcut present in existing reference-conditioned designs. A cross-context identity reward post-trains the model by scoring generated faces against held-out references of each identity, steering the model toward generating the person itself rather than copying the conditioning references. Extensive experiments demonstrate that Reprise improves identity preservation, multi-person subject binding, and naturalness, while reducing identity confusion and excessive copying of the conditioning references.

Primary area:
generative models
Keywords:
video generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WzgYOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies identity-preserving human video generation and argues that conditioning on a single reference image and scoring it with same-reference FaceSim rewards copying rather than identity. It proposes a film/TV corpus of 10K identities with image and video references from disjoint shots. The core idea is ID-RoPE, which repurposes four slow temporal RoPE planes as a Hadamard-coded identity.

Critical strengths
  1. Identity-preserving, multi-subject human video generation is a valuable task, and the observation that same-reference similarity rewards copying is well motivated. ID-RoPE is a reasonably novel design, which binds tokens to their own references through a parameter-free identity coordinate inside RoPE.
  2. The pipeline behind Reprise-10K is rigorous and reasonable (cross-episode identity clustering; references taken from shots disjoint from the target, with minimum gaps between them; yaw-balanced, diversity-aware reference selection; background removal to prevent leakage).
Critical weaknesses
  1. The task formulation and method design are incomplete and not discussed in depth.
    • The target-side identity coordinate comes from per-frame character boxes, so ID-RoPE does not appear to support inputs without layout boxes. That is the most common usage setting, and the paper does not explain how the model handles it.
    • Multiple references from different contexts carry conflicting priors, such as different hairstyles, outfits and makeup. The paper assumes these cancel out, but it does not examine which attributes the output follows or how a user can control them. This affects controllability.
  2. The experiments and evaluation are ambiguous, raising concerns about the method's actual effectiveness and the fairness of the comparison.
    • The RL reward and the final metrics appear to come from the same source. This is not a fair comparison.
    • The test set includes celebrities, and the paper does not state whether these identities appear in the training data.
    • The method introduces layout control, yet the test set, the quantitative metrics and the comparisons contain no evaluation of it.
    • Implementation details for Wan-14B are missing. While at 1.3B, the gains over same-size baselines are not significant.
  3. The qualitative evidence and dataset description are insufficient, rasing concerns about reproducibility.
    • No video files are provided, and the current visualizations are coarse with errors. The caption claims the example shows two characters of the same gender and similar age, but no such example appears in the figure. And there are no three-character visualizations.
    • There are no dataset examples or statistics.
Questions for authors
  1. Do the 100 test identities overlap with Reprise-10K? What are the results on the celebrity and ordinary-individual subsets separately?
  2. Where do the per-frame character boxes come from at test time? Which references did each baseline receive? In multi-subject evaluation, how are faces generated by baselines (which have no box input) assigned to characters?
  3. How does the method handle inputs without reference boxes? How does it handle conflicting information across multiple references?
  4. Please provide the implementation and training details for Wan-14B.
  5. Please provide more visual examples (including videos and three-character cases) and dataset details (examples and statistics).
2 comments

Response to Reviewer WzgY

1. Train/test identity overlap and celebrity/ordinary-individual subsets

The 100 test identities were fixed before constructing the training corpus. The test set includes both ordinary individuals and celebrities. During training-data collection, we excluded television series and films featuring the held-out test actors, rather than merely removing identical video clips. After data curation, we additionally performed identity-similarity retrieval and a deduplication check across the training and test character references, finding no duplicate identities. Reprise-10K and the test set therefore use an identity-level split rather than merely a clip- or shot-level split.

The test set is balanced, with 50 celebrity identities and 50 ordinary-individual identities.

2. Test-time boxes, baseline reference inputs, and multi-subject evaluation

Inference does not require user-provided target character bounding boxes or per-frame character trajectories. The boxes exported by the data construction pipeline provide target-reference correspondence guidance during SFT, rather than an additional test-time input.

In Table 1, VACE, Phantom, RefAlign, MAGREF, BindWeave, VINO, and SkyReels-V3 receive one reference image per character and no reference clips. Reprise uses two reference images and two reference clips per character by default. We will explicitly distinguish these reference inputs in the benchmark description and the Table 1 caption.

We additionally report Reprise-14B with one reference image per character and no reference clips:

Reprise-14B configurationFaceSim ↑CCIS ↑MotionSmooth ↑Natural ↑X-CLIP ↑
1 image per character; no clips0.5690.4720.9593.840.341
2 images + 2 clips per character0.5890.5080.9643.920.347

The single-image configuration exceeds the best baseline mean on four metrics and ties the best MotionSmooth score. This control matches the inference reference budget.

For multi-subject evaluation, we first detect faces in the generated video and extract their identity features. Each detected face is compared with the reference identity representations of all characters and is independently assigned to the character with the highest cosine similarity. This offline evaluation procedure is shared by Reprise and the baselines; it does not use predefined target character boxes or spatial regions and does not impose a one-to-one assignment.

FaceSim uses the conditioning references, whereas CCIS uses cross-context evaluation references of the same identity that are not provided to the generator. We will correct the original wording about the "conditioning region" to describe the actual identity-similarity-based assignment rather than a spatial-region-based assignment.

3. Inputs without reference boxes and conflicting reference information

The supported reference format consists of subject-centered face crops grouped by character. Reference assets are coded whole: every token in a character's reference images and clips shares the identity coordinate s=ns=n. No additional reference-side box annotations are required for this format.

During SFT, target tokens inside the non-overlapping region of character nn's box receive s=ns=n; tokens outside all boxes or in overlapping regions receive s=0s=0. At inference, target tokens use the default coordinate s=0s=0 without box-derived per-character assignments. The reference-side ID-RoPE identity encoding remains active. Under the fixed-carrier construction, same-character reference-reference pairs receive the identity bias +βl+\beta_l. The explicit same-identity bonus for target-reference pairs is inactive in this configuration, and target video tokens access the references through the learned content term in attention. We will clarify this stage distinction and the Table 2 ablation configurations.

For conflicting references, the task is to preserve facial identity and facial dynamics across contexts, not to reproduce all reference hairstyles, outfits, and makeup simultaneously. Cross-context pooling concerns context-dependent components in the face-embedding space; it does not mean that conflicting appearance attributes are automatically cancelled at the pixel level. References provide identity and facial-dynamics information, while text prompts specify the generation context and appearance conditions. We do not claim strict, independent control over individual appearance attributes or precise spatial layout. The revised manuscript will make this task boundary explicit.

4. Wan-14B implementation, training details, and ablation

The 14B model freezes the Wan2.1 DiT backbone and updates a LoRA adapter with rank r=128r=128 and scaling α=128\alpha=128. SFT runs on Reprise-10K for 10,000 updates at global batch size 16. Reward-gradient post-training starts from the SFT checkpoint and updates the same adapter for 200 updates at global batch size 8.

Post-training uses AdamW with a learning rate of 10−510^{-5}, a gradient-clipping threshold of 1.0, and a velocity-anchor weight of λ=1\lambda=1. RL rollouts use a 20-step Euler solver without classifier-free guidance, and the computation graph is retained only for the final denoising step. The nominal rollout budget is 200 × 8 = 1,600. Targets contain 81 frames at 480 × 832 resolution, reference images are 512 × 512 crops, and reference clips contain 65 frames.

The reward encoder is AdaFace-IR101; identity evaluation uses the AdaFace ViT-Base model provided by CVLFace, pretrained on WebFace4M. Although both use the AdaFace training objective, their backbone architectures and pretrained weights differ, and evaluation does not reuse the reward encoder. We will explicitly distinguish their roles in the revised manuscript.

We additionally report the 14B SFT-only versus Full comparison under the default two-image, two-clip configuration:

14B configurationFaceSim ↑CCIS ↑MotionSmooth ↑Natural ↑X-CLIP ↑
SFT-only0.5540.4570.9243.740.330
Full (SFT + RL)0.5890.5080.9643.920.347

Post-training increases CCIS by 0.051, approximately 11.2% relative to SFT-only, and improves the other four reported metrics. This same-scale comparison directly complements the 14B main results with evidence of the post-training contribution.

5. Additional visual results and dataset statistics

We have prepared an anonymous webpage containing four generated-video examples, each accompanied by its three reference images, and 20 dataset visualizations covering single-character and multi-character samples (one, two, and three characters). For each annotated character in the dataset examples, the webpage provides three reference images and three reference clips alongside the target video. These materials are available at the following anonymous URL:

https://review-visuals-8d3f.pages.dev/

The submitted manuscript already reports dataset statistics in Section 4.1 (page 6) and Appendix B, "Scale and scope" (page 13):

Dataset propertyAlready reported in the submitted manuscript
Number of identities10,000
Reference images per character slot3–12
Reference clips per character slot3–12
Distinct shots spanned by a reference set3–24
Target compositionSingle-character clips and co-frame clips with two or three characters

Figure 4 and Appendix B also describe the data construction pipeline, including cross-episode identity clustering, reference selection, dynamics filtering, and background removal. We will consolidate the existing statistics into a clearly labeled table and ensure that figure captions accurately describe the displayed examples.

Reviewer_6681OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reprise, a multi-reference framework for identity-preserving human video generation. The authors argue that existing methods tend to copy the appearance of reference images rather than preserve identity across different contexts. To address this, they introduce Reprise-10K (a cross-context identity dataset), ID-RoPE (an identity-aware positional encoding mechanism for multi-person generation), and a cross-context identity reward that encourages identity consistency beyond conditioning references.

Critical strengths
  1. The paper identifies an important limitation of existing methods: high reference similarity does not necessarily imply faithful identity preservation.
  2. ID-RoPE introduces identity-aware attention through positional encoding without additional learnable parameters. The approach is elegant, and the ablation results demonstrate its effectiveness in multi-person scenarios.
  3. The proposed method benefits from a dedicated cross-context dataset and multiple image/video references. It remains unclear how much improvement comes from ID-RoPE itself versus additional training data and conditioning information.
Critical weaknesses
  1. ID-RoPE assigns identity coordinates to target tokens based on per-frame character bounding boxes. However, the paper does not clarify how these bounding boxes are obtained during inference when generating videos from text and references.
  2. The cross-context reward and CCIS evaluation rely on the same face-embedding representation. Improvements may therefore partially reflect optimization toward the evaluation model rather than general identity preservation.
Questions for authors

See the Weaknesses.

3 comments

Response to Reviewer 6681

Thank you for your careful review of our work. We are pleased that you recognize its contributions, and we appreciate your constructive comments. We address your concerns below.

1. Source of per-frame character bounding boxes at inference

Inference does not require user-provided target character bounding boxes or per-frame character trajectories. The original manuscript did not clearly distinguish training-time target-reference correspondence guidance from inference-time reference-side identity grouping.

During SFT, per-frame character bounding boxes come from the annotations exported by the data construction pipeline. Target tokens inside the non-overlapping region of character nn's box receive s=ns=n, while tokens outside all boxes or in overlapping regions receive s=0s=0. Reference tokens of the same character use the same coordinate. Under the fixed-carrier construction, this provides an identity attention bonus of +βl+\beta_l for the corresponding target-reference token pairs.

At inference, target tokens are not assigned per-character identity coordinates from bounding boxes; they use the default coordinate s=0s=0. The reference-side ID-RoPE identity encoding remains active: different characters' reference tokens use different coordinates, while all reference images and clips of the same character share one coordinate. Same-character reference-reference pairs receive the identity bias, providing reference-side identity grouping.

This configuration does not provide the explicit same-identity bonus for target-reference pairs. Target video tokens instead access each character's references through the learned content term in attention. Character bounding boxes are thus training-time structural guidance rather than an additional inference input. We will clarify this stage distinction and specify the training and evaluation configurations of the Table 2 ablations in the revised manuscript.

2. Reward and CCIS face-embedding representations

The cross-context identity reward uses AdaFace-IR101, whereas identity evaluation uses the AdaFace ViT-Base model provided by CVLFace, pretrained on WebFace4M. Although both use the AdaFace training objective, their backbone architectures and pretrained weights differ; evaluation does not reuse the reward encoder. We will correct the relevant model names and explicitly distinguish their roles in the revised manuscript.

In Table 3, SFT to Full improves FaceSim from 0.502 to 0.514 and CCIS from 0.363 to 0.395. Mean MotionSmooth and X-CLIP also increase from 0.948 to 0.949 and 0.325 to 0.328, respectively. The gains are therefore not restricted to CCIS: they also appear in conditioning-reference identity similarity and the two control metrics.

Additional visual materials. We also provide additional visual results and dataset examples on our anonymous supplementary webpage. It contains four generated-video examples, each accompanied by three reference images, and 20 dataset visualizations covering one-, two-, and three-character samples.

Thank you for the rebuttal and follow-up responses. After considering the other two reviewers’ comments, I still think the paper would benefit from further revision, particularly to improve the clarity of the writing.

I found the video quality in the additional visual materials only moderately convincing. Without the original prompts, I could not assess how closely the videos followed them. The examples also did not show a clear advantage over base models with native support for reference-conditioned generation, such as MiniMax-H3.

Thank you for your reply. You are obviously very familiar with r2v task. Indeed, the paper requires further revision. We have completed the revisions of the issues currently under review and are adding user study experiments to strive for a clearer writing style in the paper. And thank you for reminding, we have updated the prompt on the webpage. https://review-visuals-8d3f.pages.dev/

Regarding the reference generation performance, yes, the effect achieved through training on a 14B model with 1w data is certainly not comparable to the current most powerful open-source base model, minimax h3. There is a significant gap in both the number of model parameters and the scale of training data, which determines the ultimate reference generation capability of the model. But our method outperforms other open-source reference models except for the base models. I believe that a valuable contribution of our work is the provision of a complete data processing pipeline (which will be open-sourced later with the dataset), ranging from internet videos to multi-role-multi-reference paired data. This has filled the gap in open-source work where high-quality data pipelines are currently absent. After all, some times data is what all we need.

Thank you again for your recognition and positive response to our work.

Reviewer_dceXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that single-reference subject-driven video generation rewards copying the reference photo rather than preserving the person. It proposes Reprise: (i) Reprise-10K, a corpus of 10K identities from TV/film with multiple reference images and clips per identity from different scenes; (ii) ID-RoPE, an extra identity coordinate in RoPE that gives same-character tokens a constant attention bonus and places references outside the target's positional range; and (iii) LoRA post-training on a frozen Wan2.1 DiT with a face-similarity reward against held-out references from other contexts. A new metric (CCIS) and a 150-case test set are introduced, and the method outperforms VACE, Phantom, MAGREF, BindWeave, VINO, RefAlign and SkyReels-V3, mainly at 14B and on multi-subject cases.

Critical strengths
  • Clear and sensible framing: identity should be what stays constant across contexts. Reporting FaceSim and CCIS side by side is a useful diagnostic.
  • ID-RoPE is simple and parameter-free, with a clean closed-form bias, and the ablation shows it matters where expected (multi-subject).
  • Reward-target ablation (Table 3) and the token-matched static-repeat control (Table 4) are careful experiments.
Critical weaknesses
  • Reward and metric use the same face model (AdaFace). The model is optimized on the metric it is evaluated with, so CCIS gains may be reward hacking. No independent recognizer and no human study.
  • Unfair baseline comparison. Test gives 2 images + 2 clips per character, but most baselines are single-image and the paper never says what they receive. Baselines are also not fine-tuned on Reprise-10K, so data vs. method is not separated.
  • Inference needs per-frame bounding boxes for each character. This is a strong extra input no baseline uses, not discussed as a limitation, and robustness to bad or missing boxes is untested.
  • 14B carries the story but has no ablations or training details. At 1.3B gains are small (CCIS 0.395 vs. 0.383); the big jump exists only at 14B as a single number.
  • Small self-built benchmark, no variance reported, several gaps within ~0.01. Test identities are web celebrities while training uses actors; train/test identity overlap is not checked.
  • Copyrighted footage of real people, no ethics/consent discussion, no release commitment for data, benchmark, or code.
Questions for authors
  1. What exact input does each baseline receive in Table 1?
  2. Does CCIS hold with a face recognizer different from the reward model? Can you add a human study?
  3. How are per-frame boxes obtained at test time, and how does performance degrade with coarse or missing boxes?
  4. Can you provide SFT-only vs. full ablation and training details at 14B?
  5. Did you check identity overlap between the test set and Reprise-10K?
  6. Will data, benchmark, and code be released, and under what license?
2 comments

Response to Reviewer dceX

Thank you for your careful review of our work. We are pleased that you recognize its contributions, and we appreciate your constructive comments. We address your concerns below.

1. Reward model and identity evaluation

We agree that evaluation with a face recognizer distinct from the reward network strengthens the evidence. The cross-context identity reward uses AdaFace-IR101, whereas identity evaluation uses the AdaFace ViT-Base model provided by CVLFace, pretrained on WebFace4M. Although both use the AdaFace training objective, their backbone architectures and pretrained weights differ; identity evaluation does not reuse the reward encoder. The original manuscript did not clearly distinguish these encoders. We will correct the model names and specify their respective roles in the revised manuscript.

In Table 3, SFT to Full improves FaceSim from 0.502 to 0.514 and CCIS from 0.363 to 0.395. The mean MotionSmooth and X-CLIP scores also increase from 0.948 to 0.949 and 0.325 to 0.328, respectively. The improvements are therefore not confined to CCIS.

Within the limited rebuttal period, we have received five human-evaluation responses. The preliminary aggregate gives Reprise the highest scores in visual consistency, naturalness, and prompt following. We are continuing to collect feedback and will update the human-study results in a subsequent version.

2. Reference inputs and baseline comparison

In Table 1, VACE, Phantom, RefAlign, MAGREF, BindWeave, VINO, and SkyReels-V3 receive one reference image per character and no reference clips. Reprise uses two reference images and two reference clips per character by default. The original benchmark description did not clearly distinguish the reference inputs actually supplied to each method. We will correct the benchmark description and the Table 1 caption.

We additionally report a Reprise-14B control using only one reference image per character and no reference clips:

Reprise-14B configurationFaceSim ↑CCIS ↑MotionSmooth ↑Natural ↑X-CLIP ↑
1 image per character; no clips0.5690.4720.9593.840.341
2 images + 2 clips per character0.5890.5080.9643.920.347

The single-image configuration exceeds the best baseline mean for FaceSim, CCIS, NaturalScore, and X-CLIP, and ties the best baseline on MotionSmooth. The default multi-reference configuration further improves all five reported metrics at 14B.

Appendix E, Table 4, already provides the reference-conditioning ablation at 1.3B, including different reference budgets and the token-matched static-repeat control. Its default two-image, two-clip row matches the Reprise-1.3B results in Table 1. The additional single-image control matches the inference reference budget; the contributions of ID-RoPE and the reward design are assessed through within-framework ablations.

3. Per-frame character bounding boxes at inference

Inference does not require user-provided target character bounding boxes or per-frame character trajectories.

During supervised fine-tuning (SFT), per-frame character bounding boxes exported by the data construction pipeline determine the target-side identity coordinates. A target token inside character nn's box, excluding overlapping regions, receives s=ns=n; tokens outside all character boxes or in overlapping regions receive s=0s=0. Reference tokens of the corresponding character use the same identity coordinate, providing the same-identity attention bonus for target-reference pairs. These annotations provide training-time structural guidance for identity correspondence rather than an input required from the user at deployment.

At inference, target tokens are not assigned per-character identity coordinates from bounding boxes; all target tokens use the default coordinate s=0s=0. The reference-side ID-RoPE identity encoding remains active: different characters receive different identity coordinates, while every token in the reference images and clips of the same character shares the same coordinate.

Under the fixed-carrier construction in Eq. (4), same-character reference-reference pairs receive the identity bias +βl+\beta_l, whereas the identity term is zero across characters. The explicit same-identity bonus for target-reference pairs is inactive in this configuration; target video tokens use the learned content term in attention to access the reference information.

Training-time explicit target-reference correspondence and inference-time reference-side identity grouping are therefore two distinct uses of the identity coordinate. We will make the stage-specific coordinate construction and the training and evaluation configurations of the Table 2 ablations explicit in the method description and table caption.

4. 14B SFT-only versus Full and training details

We additionally report the 14B SFT-only versus Full comparison under the default two-image, two-clip configuration:

14B configurationFaceSim ↑CCIS ↑MotionSmooth ↑Natural ↑X-CLIP ↑
SFT-only0.5540.4570.9243.740.330
Full (SFT + RL)0.5890.5080.9643.920.347

Post-training increases CCIS by 0.051, approximately 11.2% relative to SFT-only, and improves the other four metrics. This same-scale comparison directly complements the 14B main results with evidence of the post-training contribution.

The training budgets for both model scales are:

Model scaleSFT updatesSFT global batch sizeRL updatesRL global batch size
1.3B10,000162008
14B10,000162008

Both stages freeze the Wan2.1 DiT backbone and update the same LoRA adapter, with rank r=128r=128 and scaling α=128\alpha=128. Reward-gradient post-training uses AdamW with a learning rate of 10−510^{-5}, a gradient-clipping threshold of 1.0, and a velocity-anchor weight of λ=1\lambda=1. RL rollouts use a 20-step Euler solver without classifier-free guidance, and the computation graph is retained only for the final denoising step. The nominal RL rollout budget is 200 × 8 = 1,600 per model scale.

5. Identity overlap between Reprise-10K and the test set

The test identities were fixed before constructing the training corpus. They include ordinary individuals and a subset of celebrities. During training-data collection, we excluded television series and films featuring these held-out test actors, rather than merely removing identical video clips.

After data curation, we additionally performed identity-similarity retrieval and a deduplication check across the training and test character references, finding no duplicate identities. Reprise-10K and our test set therefore use an identity-level split rather than merely a clip- or shot-level split.

The benchmark contains 100 identities and 150 cases: 100 single-subject cases, 30 two-subject cases, and 20 three-subject cases, with three generation seeds per case. We will clarify the case composition and evaluation scope in the revised manuscript.

6. Release plan, licenses, and data rights

The training code, complete data construction pipeline, and curated Reprise-10K training dataset and test dataset are ready for release. We will release them immediately upon acceptance.

Our original training code and data construction pipeline will use Apache-2.0; third-party code and pretrained weights retain their original licenses. Access to the training and test data will be provided under a research-only agreement, to the extent permitted by source-material copyright and the rights of the depicted individuals. For material that cannot be publicly redistributed, we will provide source indices, annotations, and construction procedures instead. Research-only terms do not replace the underlying rights authorizations. We will add the corresponding data-access and rights information to the revised manuscript.

Additional visual materials. We also provide additional visual results and dataset examples on our anonymous supplementary webpage. It contains four generated-video examples, each accompanied by three reference images, and 20 dataset visualizations covering one-, two-, and three-character samples.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.