- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes StreamingSwap, a real-time streaming video face swapping framework built on DreamID-V. It introduces three components: a Joint Rec-ID Loss that adds noise-dependent identity supervision, Face-RoI Generation that processes only face-region tokens, and Short-Context Rollout Training that adapts Self Forcing for chunk-wise streaming. The authors also build HumanFaceVid, a 1,000-hour dataset derived from OpenHumanVid. The method runs at about 20 FPS on a single RTX 4090 and handles videos up to 10 minutes.
Critical strengths
- The paper targets a practical problem, achieving real-time reference-guided face swapping (about 20 FPS) on a single consumer GPU.
- The appendix includes fairly extensive ablations on loss weighting, denoising steps, and window sizes.
Critical weaknesses
- The designs of this paper are not very impressive. Face-RoI Generation resembles existing mask-aware token selection methods (e.g., YOSE, RegionE, SpotEdit), and the rollout training is a direct adaptation of Self Forcing. These related works are not discussed.
- The results in experimental section are not very strong. Identity similarity (0.577) is far below InSwapper and DreamID-V (~0.73), and the attribute metrics seem to reward weaker swaps. All results are on the authors' own benchmark without a user study.
Questions for authors
- The identity loss is computed on the clean estimate at all noise levels. At high noise, the decoded frames are likely blurry and ArcFace embeddings may be unreliable. Have you examined whether the identity loss gives meaningful gradients in this regime? Also, why use a hard step function for λ_rec instead of a continuous schedule?
- Restricting tokens to the face region removes surrounding context such as hair, neck, and scene lighting. How do you handle boundary consistency (e.g., hairline, skin tone at the blending edge), and how sensitive are the results to the dilation size?
- How does Face-RoI Generation handle multiple faces, large head motion where the RoI changes size across frames, and occlusions crossing the RoI boundary?
- Training uses only five autoregressive windows, while inference runs for up to 10 minutes. Why is this enough to prevent error accumulation over hundreds of windows? Did you try longer rollouts during training?
- You replace the distribution-matching objective in Self Forcing with the Phase I reconstruction and identity losses. What motivates this choice, and how does it compare to using DMD?
- The latent MSE in Sec. 3.1 is computed between real videos and InSwapper outputs, which also contain synthesis artifacts. How do you separate the effect of identity change from InSwapper artifacts in this analysis?
2 comments
Response to Q4: Short Training Rollouts and Long-Duration Stability
Five-window training exposes the model to its generated histories. During inference, the context remains bounded, while the target video continuously supplies appearance and motion cues and the source image provides a persistent identity reference. This reduces dependence on generated history for maintaining long-term content.
We have added duration-resolved evaluation of five outputs lasting 3.5–19 minutes, reporting identity and attribute trajectories in 30-second intervals, together with changes relative to the first-minute baseline. The curves show no consistent progressive deterioration across the evaluated cases (Appendix I and Figs. 20–21, p. 27–28). These results support the chosen rollout length empirically; they do not guarantee stability for arbitrary durations. The current study does not include a controlled comparison with longer training rollouts.
Response to Q5: Direct Supervision Instead of DMD
Our Phase-I model already supports competitive four-step generation (Table 5, p. 24), and our paired data provide real-video reconstruction targets and source-identity supervision. We therefore adapt the model through generated-history rollouts using these existing objectives, without introducing an additional DMD stage.
This choice directly supervises the task requirements while addressing the training–inference history mismatch. Its benefits are demonstrated by the self-rollout ablation (Table 4 and Fig. 8, p. 10). We clarify this distinction from distribution-matching approaches in §4.3 (p. 7) and Appendix A.3 (p. 16).
Response to Q6: Identity Changes Versus Synthesis Artifacts
We selected qualitative cases without visible synthesis artifacts and used the same InSwapper pipeline across comparisons to control the synthesis procedure. The self-swap control provides a reference for pipeline-induced deviations, while the identity-preserving mild-blur example contextualizes the latent-error scale (§3.1 and Fig. 2(a), p. 4–5). The training ablation provides independent supporting evidence: removing explicit identity supervision substantially weakens identity transfer (Table 4, p. 10).
Thanks for your careful reading and constructive feedback. We have revised the manuscript to address your concerns, with changes and additions marked in blue. Below, we provide point-by-point responses, including references to the corresponding sections, figures, tables, and pages in the revised manuscript.
Response to W1: Relation to Prior Editing and Rollout Methods
We acknowledge that regional computation and training on generated histories build on prior work. We have added discussions of YOSE, RegionE, SpotEdit, and Self Forcing in Appendix A.3 (p. 16) and explicitly acknowledged their influence in §4.2–4.3 (p. 6–7).
Our contribution is the task-specific analysis and resulting adaptation: we establish that reference-guided face swapping can substantially reduce spatial and temporal context because the target video already supplies detailed appearance and motion cues. Face-RoI omits outside-region Transformer computation without an additional background-context mechanism, while short-context rollout retains within-window bidirectional attention and uses direct paired supervision without an additional DMD stage. Their practical benefits are supported by Table 4 and Fig. 8 (p. 10).
Response to W2: Identity Transfer and Human Perceptual Evaluation
We agree that attribute metrics alone can favor insufficient identity changes. Indeed, removing identity supervision improves attribute scores but reduces ID Sim. from 0.573 to 0.241, illustrating precisely this trade-off (Table 4, p. 10).
To evaluate whether our complete method achieves convincing identity transfer, we have added a user study comprising 145 participant–case evaluations. Our method receives the highest mean ratings among the three evaluated methods for identity resemblance, attribute preservation, and video quality: 4.05/4.21/4.15, compared with 3.97/2.63/2.84 for DreamID-V and 2.98/3.19/2.44 for InSwapper. Please see Table 3 and Fig. 7 (p. 9), with the protocol in Appendix D (p. 23).
The added real same-identity analysis also illustrates how appearance variation affects ArcFace similarity, explaining why higher embedding similarity alone does not establish better perceptual identity transfer (§5.1 and Fig. 5, p. 9).
Response to Q1: High-Noise Identity Supervision and the Weighting Schedule
The denoising-trajectory analysis shows that substantial identity information emerges within the early sampling steps: the decoded clean estimates need not remain uninformative simply because the current latent has high noise (Fig. 16, p. 24). Moreover, removing identity supervision from either noise interval weakens final identity transfer, supporting the usefulness of retaining it throughout training (Appendix F, Figs. 17–18, p. 25).
We use a step function as a minimal, interpretable parameterization for examining how reconstruction and identity supervision should be balanced across noise regimes. Comparisons with constant weighting and alternative configurations, together with the threshold sweep, support its effectiveness (Fig. 17, p. 25–26). Continuous schedules are compatible with our formulation; we do not claim that the step function is uniquely optimal.
Response to Q2: Boundary Consistency and RoI Dilation
Our computation region is an enlarged facial region, retaining nearby context rather than tightly cropping the face boundary. Selected tokens preserve their original positional encodings, and the predicted latents are scattered back and smoothly blended with target-video latents using a feathered mask (§4.2, Eq. 10, p. 6).
We have added a dilation-sensitivity experiment over 32–128 pixels, keeping the blending parameters fixed. Identity and attribute metrics vary modestly; increasing the radius from 64 to 128 pixels raises token coverage from 38.53% to 68.84% without consistent quality gains. This supports the default radius as a practical quality–computation trade-off. Please see Appendix H and Table 8 (p. 26–27).
Response to Q3: Multiple Faces, Head Motion, and Occlusion
The per-frame face mask specifies the selected person and tracks changes in the editing region, accommodating head motion and varying RoI size. Tokens retain their original spatiotemporal positions, while padding and attention masking accommodate different sequence lengths during batched training (§4.2, p. 6). In multi-person scenes, this mechanism selects the person to edit. For occlusions, the model uses target-video conditioning together with pretrained diffusion priors, while the enlarged RoI and feathered blending support integration with surrounding content. Qualitative examples illustrate multiple faces, challenging motion and occlusion handling in Fig. 4 (p. 8) and Appendix C, Figs. 11–13 (p. 18–21).