acceptodds
Under review as a conference paper at ICLR 2027

FreeCHROMA: Generative Chroma Keying with Flexible Background References

Abstract

In film production, chroma keying composites an approved foreground performance into a new environment, but conventional workflows require coordinated background plates and dedicated harmonization. Recent generative methods often require pre-aligned backgrounds, focus on background-conditioned foreground relighting, or represent the environment only through an edited first frame. These restrictions limit background selection, scene composition, and camera motion while adding preparation or editing effort. We present FreeCHROMA, which adapts a pretrained video diffusion model in a parameter-efficient manner for generative video compositing from one to three unaligned background references. The model integrates the foreground into the referenced environment, preserving foreground identity, motion, and fine visual details while generating a spatiotemporally coherent background. Foreground–background harmonization occurs within the same generation process, without a separate relighting stage. Optional text instructions help disambiguate complex spatial relationships and camera behavior. To train the model, we construct pseudo-paired data from readily available real monocular videos through foreground–background separation and augmentation with existing foundation models. The original videos serve as training targets, while automated processing and quality verification enable scalable data construction. Across reference fidelity, foreground preservation, scene integration, harmonization, and overall video quality, FreeCHROMA consistently outperforms recent compositing and conditional video-generation baselines, with particularly clear gains under camera motion. FreeCHROMA relaxes the constraints of existing tools to enable end-to-end generative chroma keying from unaligned inputs, with exploratory experiments demonstrating its potential in robotics.

Primary area:
generative models
Keywords:
Generative Chroma Keying, Background Replacement, Video Editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Zac2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FreeCHROMA performs generative video compositing from a masked foreground video, one to three spatially unaligned background references, and optional text. It adapts a pretrained video diffusion model through foreground-and-mask conditioning, persistent background-reference tokens, and LoRA. Its main contribution is a pseudo-pair construction pipeline that perturbs foreground illumination and background viewpoints while retaining the original video as the training target. Experiments evaluate general video quality, VLM-assessed compositing quality, estimated camera trajectories, and the effects of the training augmentations. Qualitative robotics applications are also presented.

Critical strengths
  1. The supervision pipeline addresses identifiable training shortcuts. Foreground illumination perturbation discourages direct copying, while background viewpoint augmentation discourages treating reference images as aligned target frames. The ablations provide evidence that these choices improve harmonization and camera-trajectory consistency.

  2. Joint composition and harmonization offer a potentially useful workflow. The model preserves the supplied foreground performance while generating its surrounding environment, reducing the need for separately prepared background sequences and subsequent appearance adjustment.

Critical weaknesses
  1. The necessity of a dedicated method and its methodological novelty are not sufficiently established. The architecture primarily combines established conditioning mechanisms and LoRA, with the main contribution residing in data construction. Although unaligned references offer convenience, the paper does not establish how often this constraint is a decisive bottleneck or whether it requires specialized training. A plausible alternative is to use an image-editing model, potentially coordinated by a VLM agent, to prepare an aligned and harmonized first frame or keyframes, followed by an existing compositing or propagation method such as FlowPortal where compatible. Such a pipeline may have limitations under camera motion, but a controlled comparison is needed to establish the practical advantage of learning composition directly. Without this evidence, the contribution appears relatively incremental for a specialized application.
Questions for authors

1.Contribution beyond data and fine-tuning: Under the same backbone, curated training pairs, and optimization budget, how does FreeCHROMA compare with a simple video-editing conditioning formulation? 2. Generalization from reconstruction to cross-environment composition: Training reconstructs the original video from foreground and background conditions derived from that same video, where placement, support, and occlusion relationships are already compatible. How reliably does the model generalize when a new environment requires different foreground–background relationships, such as new occlusions or contact with different support surfaces? Can you provide a targeted evaluation separating these cases from appearance harmonization? 3. Comparison with staged workflows: The paper includes ID-V2V + FLUX.2 as a first-frame-editing baseline. How are its edited frames constructed and checked for composition quality? To what extent do the observed differences arise from initial-frame preparation versus limitations of subsequent video propagation, particularly under substantial camera motion? 4. Scope beyond foreground-preserving composition: Which components of the conditioning and supervision pipeline can extend to broader video editing tasks, such as material changes, object replacement, or structural scene edits? Which depend specifically on preserving the supplied foreground performance?

3 comments

We thank the reviewer for recognizing how our generative chroma keying approach, with persistent conditioning from unaligned environment references, can support creative workflows, and for acknowledging our pipeline for deriving scalable supervision from real monocular videos. We also appreciate the recognition of our design for addressing training shortcuts, the supporting ablation evidence, and the practical value of joint composition and harmonization. Below, we address the concerns and questions point by point.

Concern 1: Methodological novelty

Thank you for raising this concern. We would like to clarify that our contribution is to efficiently adapt a pretrained I2V model to natively use unaligned environment images as persistent references for generative chroma keying. In creative workflows, foreground footage and environment assets are often acquired separately, so their alignment cannot be assumed. This requires training the model to infer coherent foreground–background relationships from references that do not prescribe the target viewpoint or layout, while preserving foreground fidelity.

Our supervision and data construction pipeline are specifically designed to teach this capability. Since paired examples of unaligned inputs and coherent target videos are difficult to collect naturally, we construct scalable pseudo-paired supervision in reverse from real monocular videos.

Concern 2: Comparison with a simple video-editing conditioning formulation

We thank the reviewer for raising this question. To assess the capabilities of state-of-the-art open-source video editing models on our task, we included VACE and Bernini as baselines in Tabs. 1–3. Both use Wan-family diffusion backbones in the 14B class, making them comparable to FreeCHROMA in backbone family and scale. This supports the fairness of the comparison in these respects, although their training data and optimization budgets are not matched.

Our pseudo-paired data are not directly compatible with the default training pipelines of VACE and Bernini. Retraining these models on our dataset would require adapting their data representations and conditioning interfaces. We therefore evaluate their released pretrained models using their default inference pipelines.

Fig. 4 illustrates foreground-preservation failures, including changes in the two characters’ appearance and identity in the middle example, and foreground scale drift in the right example. These results show that the evaluated general editing pipelines do not consistently meet the foreground-fidelity requirements of this task, motivating our task-specific conditioning and supervision.

Concern 3: Generalization from reconstruction to cross-environment composition

We thank the reviewer for raising this concern. As described in Sec. 4.1, the qualitative results in Fig. 4 and quantitative results in Tabs. 1–3 already evaluate cross-environment composition: each foreground video is paired with two different sets of target background references. The Scene Integration metrics in Tab. 2 also assess foreground–background spatial compatibility and illumination harmony within the generated scene through VLM scoring.

Learning compatible object placement, support, and occlusion relationships is a central motivation for our use of unaligned background references. As described in the main paper, these references provide environment information without prescribing a composition aligned with the supplied foreground performance. When multiple arrangements are plausible, the optional text prompt further specifies the intended spatial and interaction relationships, guiding the model toward a particular composition. During training, we retain the original real monocular video as a coherent target and construct illumination-perturbed foreground inputs and unaligned background references from it. Reconstruction thus provides supervision for learning foreground–background composition and harmonization under unaligned reference conditioning.

Occlusion and contact relationships are difficult to evaluate reliably in open-ended composition, where multiple spatial arrangements may be valid and dedicated evaluation criteria are needed. We therefore assess overall scene integration through VLM scoring, without separately reporting success rates for these relationships.

Concern 4: Comparison with staged workflows

Thank you for this suggestion. Tables 1–3 include two staged baselines, ID-V2V + Paste and ID-V2V + FLUX.2. Both prepare a composite first frame and then use ID-V2V for video propagation, following the first-frame preparation and propagation workflow discussed by the reviewer.

To further directly address the reviewer’s suggestion of multi-keyframe editing “potentially coordinated by a VLM agent,” we have added additional qualitative comparisons in Figs. R1 and R2 (in page 10) of the revised PDF. We use both GPT-Image (Agentic with GPT Astra as the VLM), a state-of-the-art closed-source agentic image-editing baseline, and FLUX.2, a state-of-the-art open-source image-editing model, to edit multiple keyframes. The edited keyframes are then propagated into videos using FlowPortal (Fig. R1) and ID-V2V (Fig. R2).

These examples illustrate a consistency challenge in multi-stage editing-and-propagation pipelines. Since the same foreground and environment can admit multiple plausible compositions, separate keyframe edits may select different background layouts. Even when individual edits appear plausible, they can provide inconsistent scene constraints for subsequent video propagation, resulting in layout drift and hallucinated transitions, including objects appearing or disappearing. The recurrence of these failures across both editing models and propagation methods highlights a system-level limitation of the tested pipelines: strong standalone image-editing performance alone does not guarantee consistent keyframe references or coherent videos.

FreeCHROMA jointly generates the video using foreground conditioning and persistent context from the original unaligned environment references. This formulation retains direct access to the original scene information throughout generation and avoids introducing conflicting constraints through intermediate keyframe edits. The comparisons illustrate its practical benefit when independently edited keyframes fail to provide consistent scene references.

Concern 5: Scope beyond foreground-preserving composition

Thank you for highlighting this broader perspective. We agree that the underlying paradigm can be extended to a wide range of tasks beyond foreground-preserving composition. More generally, a mask specifies the content to retain, references guide the generation of the remaining regions, and the resulting video is jointly harmonized. This provides a flexible formulation for tasks such as reference-guided video editing, content replacement, scene recomposition, and data augmentation. The robotics examples in Sec. 4.6 illustrate several applications of this broader paradigm. In this paper, we focus on generative chroma keying—background composition and harmonization—as a concrete and practically relevant instantiation that remains challenging even with state-of-the-art general-purpose image-editing models. Extensions to other tasks can build on the same formulation with appropriate masks, references, and task-specific training data.

We thank the authors for their response. However, given the somewhat limited novelty and questionable practical applicability of the proposed method, I maintain my original rating.

Reviewer_1VH6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FreeCHROMA, a method for generative chroma keying. Given a foreground video with its mask, 1- 3 background reference images that are not aligned with the video, and optional text, it generates a video that places the foreground in the referenced environment while keeping the original performance. The main contribution is a data construction method that builds pseudo-pairs from natural videos. The original video is used as the target, and two augmentations are applied only on the input side: illumination perturbation of the foreground to prevent simple copying, and viewpoint changes of the foreground-removed background to prevent simple pasting. The model adapts Wan2.2-Fun-A14B-Control with LoRA. On 1,000 test cases, it is compared with seven baselines and achieves the best results on all VLM-based and camera-trajectory metrics, and the best or second-best results on six of seven VBench metrics.

Critical strengths
  • Practical task setting and scalable supervision. The method doesn't require pre-aligned backgrounds or edited keyframes, directly addressing real constraints in film production. Large-scale training data can be built from natural videos without green-screen capture, and the method trains only LoRA adapters, added input-projection weights, and a background-reference convolution, making adaptation parameter-efficient.
  • Consistent improvements and clear ablations. The method outperforms seven baselines from different method families on all VLM-based and camera-trajectory metrics and ranks first or second on six of seven VBench metrics. Removing each augmentation leads to the expected failure, clearly showing each component's role.
Critical weaknesses
  • Training and inference conditions do not match. Background references are taken from the same video as the foreground, so the model never sees a cross-scene pair during training. The illumination perturbation applies smooth, clip-constant transforms (global color changes, low-frequency gain fields, and normal-based shading) on top of the original appearance, and does not remove the original shadows, highlights, or local shading. The model may therefore infer the scene from the residual foreground lighting instead of relighting the foreground from the reference. The ablation (Tab. 4) does not rule this out.
  • The evaluation does not measure the main claims. No metric measures reference fidelity, which the abstract lists as a key result. The VLM evaluator is not shown the reference (Appendix C), and its FG Preservation score is saturated (9.904 vs. 9.901 for ID-V2V + FLUX.2). Tab. 3 measures how well the source camera motion is reproduced, not adherence to the reference. Without a baseline on the same backbone, it is also unclear whether the gains come from the data recipe or from Wan2.2-A14B.
Questions for authors
  • Do the test background references come from the same scene as the foreground or from different scenes? If different, how were the pairs chosen?
  • What happens when the capture conditions of the foreground and background are very different (e.g., a person filmed indoors placed on a beach)? Can the model change the foreground lighting to match the reference environment, or does it generate a background that matches the foreground lighting? Since this directly relates to Weakness 1, please discuss it with concrete examples.
  • How does performance change with the number of references K (1 to 3) and with or without text? Since the model is trained with dropout, they should be able to evaluate this without retraining.
  • Are there results comparing other conditioning designs on the same backbone?
4 comments

We thank the reviewer for the constructive feedback. Below, we address the concerns with clarifications and additional quantitative and qualitative results included in the revised PDF.

Concern 1: Training–inference mismatch and illumination generalization

Thank you for raising this concern. To directly address it, we have added Fig. R3 (p. 10) of the revised PDF, using the same foreground input with references depicting substantially different lighting conditions, including nighttime sodium-lamp lighting, spotlights, blue hour, golden-hour backlighting, and overcast skylight. These results demonstrate that our model can effectively harmonize foreground illumination across a wide range of target environments.

Our main evaluation also explicitly tests cross-scene composition. The background reference images are generated using an image generation model and depict environments different from the foreground source scenes. Each of the 500 foreground videos is paired with two sets of these generated background references, yielding 1,000 cross-scene test cases. All methods are evaluated on the same foreground–background pairs.

We intentionally use same-source videos for training because they provide real, temporally coherent targets with naturally harmonized foreground and background. For a cross-scene input pair, a ground-truth video showing the same foreground performance in the new environment is generally unavailable. We perturb foreground illumination and background-reference viewpoints only on the input side, while keeping the target video unchanged. This provides reconstruction-based supervision under mismatched input conditions, designed to teach the model to reconcile foreground and environment cues while preserving the original performance. Our method therefore learns from same-scene supervision, with cross-scene generalization explicitly assessed by the evaluations above.

For foreground illumination, we deliberately use analytical illumination perturbations, including normal-guided shading. We agree that these perturbations do not remove every original shadow, highlight, or local shading cue. This reflects an intentional trade-off: generative relighting could produce larger illumination changes, but may also introduce unintended changes to foreground identity and fine visual details. During adaptation, we therefore prioritize precise foreground preservation over maximizing the range of illumination changes covered by the training perturbations. The pretrained video model already favors foreground illumination that is consistent with the environment, further supporting this design choice.

Concern 2: Reference fidelity and interpretation of the evaluation metrics

Thank you for pointing out that our original evaluation did not clearly separate these claims. To assess the background itself, we conducted a new 500-video background reconstruction experiment. The table of comparison result is shown below and we have also added the evaluation protocol and results to Appendix F (Table 6, p. 19) of the revised PDF, with the new content highlighted in yellow. FreeCHROMA achieves an original-background agreement score of 9.24, the highest among all compared methods, compared with 8.80 for the strongest baseline.

MethodOriginal-background agreement (VLM) ↑
Lumen5.70
VACE-14B (Wan)7.96
IllumiCraft7.63
FlowPortal6.86
ID-V2V + Paste7.35
ID-V2V + FLUX.28.80
Bernini8.51
FreeCHROMA9.24

In this experiment, the model receives the masked foreground video and an unaligned reference image of its source scene. The original, unmasked video is reserved as ground truth. This complementary setting differs from the 1,000-case cross-scene benchmark and provides a known background target for evaluation.

We ask Qwen3-VL to compare six time-matched pairs of original and generated frames, judging only the visible background. Scores range from 1, indicating unrelated backgrounds, to 10, indicating faithful scene layout, distinctive details, and appearance across the sampled frames. The evaluation allows rendering differences but penalizes missing or rearranged scene content and backgrounds that fail to evolve consistently with the camera.

We agree that the original VLM scores in Table 2 assess foreground preservation and scene integration, while Table 3 measures agreement with the source camera motion. Neither directly measures reference fidelity. The new experiment provides a complementary measure of background reconstruction under unaligned reference conditioning.

Concern 3: Attribution of performance gains

Thank you for raising this point. Our evaluation already includes several baselines built on the Wan 14B family, including VACE, ID-V2V, and Bernini. In particular, Bernini also uses Wan2.2-A14B. FreeCHROMA outperforms these methods across foreground preservation, background fidelity, foreground–background integration, and illumination harmony, demonstrating its advantages for generative chroma keying even against methods built on strong Wan backbones.

These results support the effectiveness of our task-specific formulation, which combines persistent environment-reference context with targeted data construction and training to jointly preserve the foreground, generate backgrounds faithful to the references, and harmonize their appearance. Furthermore, our ablations in Table 4 keep the backbone, conditioning architecture, training data, and training schedule fixed, directly demonstrating the contribution of the proposed training components. Together, these comparisons and controlled ablations substantiate the benefits of our design and training strategy for this task.

Regarding input configurations, conditioning dropout supports K = 1–3 references and optional text within the same trained model. Our core contribution is the formulation and targeted training strategy for generative chroma keying. A systematic ablation of these input configurations is beyond this central scope and is not essential to establishing these contributions; we therefore focus our experiments on validating the proposed design and training components.

Figs. R1 and R2 (in page 10 of the revised PDF) further provide qualitative comparisons with staged pipelines that edit multiple keyframes using GPT-Image (Agentic) or FLUX.2 and propagate them using FlowPortal (Fig. R1) or ID-V2V (Fig. R2). These examples show background distortions, layout drift, and disappearing or hallucinated objects in the staged outputs, while FreeCHROMA maintains more coherent scene structure. These comparisons provide qualitative support for our formulation, which retains persistent environment-reference context throughout joint video generation.

Dear authors, thank you for your reply and the additional results. My concerns have been adressed, I have thus increased my score.

Dear Reviewer,

Thank you very much for your positive feedback! We are glad that our response and additional results have addressed your concerns. Your constructive suggestions have helped us improve the paper.

Reviewer_EbbwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies chroma keying, which aims to composite a foreground performance video into a new environment. Existing generative approaches either require backgrounds that are pre-aligned with the foreground, focus mainly on foreground relighting conditioned on the background, or represent the target environment using only an edited first frame, which limits scene and camera motion.

The paper proposes FreeCHROMA, a parameter-efficient framework built on a pretrained video diffusion model that composites a foreground video into a new environment using 1–3 unaligned background reference images. The method aims to preserve foreground identity, motion, and visual details while establishing appropriate spatial relationships with the background and harmonizing foreground/background illumination.

A key challenge is the lack of paired training data. To address this, the authors construct pseudo-paired training data from real monocular videos using foreground-background decomposition and several data augmentation strategies. On a 1K-video evaluation set constructed from HumanVid-Real and OpenVidHD, FreeCHROMA outperforms most baselines in overall video quality measured by VBench, while achieving performance comparable to Bernini. It also performs the best in camera-trajectory consistency and obtains the best VLM-based scores for foreground quality and scene integration. Ablation studies demonstrate the importance of foreground augmentation and viewpoint augmentation. The paper further presents potential applications in robotics.

Critical strengths
  • The task studied in this paper has clear practical value, as compositing a foreground performance video into a new environment is relevant to content creation, virtual production, and related applications.

  • The paper identifies training-data construction as a key challenge for this task and proposes a scalable pseudo-paired data construction pipeline, which is valuable for scaling this problem to larger datasets.

Critical weaknesses
  • Missing related work. The paper should discuss ActAnywhere: Subject-Aware Video Background Generation [NeurIPS 2024], which is closely related to the problem setting.

  • Figure 3 is somewhat confusing. The foreground video and the corresponding target video do not appear to match well. Please revise the figure.

  • It is unclear how the two different sets of unaligned background references in the paper are obtained. In addition, I would like to see several examples where the same foreground video is combined with different sets of background references, which would better demonstrate how effectively the method follows the reference environment.

  • In Table 1, the specific versions of the compared models should be reported where applicable, e.g., whether the method uses a 1.3B or 14B variant.

  • Section 4.1 states that the evaluation videos range from static scenes to scenes with large camera motion. However, the exact distribution is not reported. This is important because static examples are generally easier and may result in lower errors in Table 3. Please report the proportion of static and large-motion cases, or otherwise provide a more detailed breakdown of camera motion.

  • It is unclear why RPE is not reported in Table 4.

  • The main technical contribution appears to rely heavily on the data augmentation strategies used during pseudo-paired data construction, particularly lighting augmentation and viewpoint augmentation. However, the paper does not clearly explain whether these augmentation strategies introduce any methodological novelty compared with existing data-construction pipelines in related areas such as video editing, video relighting, or foreground-background compositing. The authors should better clarify what is technically new in the proposed data-generation pipeline beyond adapting existing augmentation techniques to this task.

  • Since training-data construction is one of the core contributions, several implementation details should be described more precisely. For example, the paper states that “clips with unstable masks, severe blur, or excessive foreground occupancy are rejected,” but does not clearly define the thresholds or criteria for unstable, blur, and excessive. Similarly, it is unclear how keyframes are selected based on the estimated camera trajectory. The statement that “the VLM rejects candidates that alter scene identity or introduce conspicuous artifacts” also needs more implementation details, including the prompt, decision criterion, and whether the filtering is fully automatic.

Questions for authors
  • How are the two sets of unaligned background references constructed or selected? Can the authors provide examples where the same foreground video is composited with several different background-reference sets?

  • What is the distribution of camera-motion magnitude in the 1K-video evaluation benchmark? In particular, what fraction of the test set corresponds to static and large camera motion?

  • Why is RPE omitted from Table 4?

  • What is technically novel about the proposed lighting and viewpoint augmentation strategies compared with data-construction pipelines used in related tasks such as video editing, video relighting, or foreground-background compositing?

  • Please provide more details about the pseudo-paired data construction pipeline. How are unstable masks, severe blur, and excessive foreground occupancy quantitatively defined? How are keyframes selected from the estimated camera trajectory?

  • How exactly is the VLM-based filtering implemented? What prompt and decision rule are used to determine whether a candidate changes scene identity or introduces conspicuous artifacts?

  • Which exact model variants are used in Table 1, especially for methods with multiple parameter scales such as 1.3B and 14B?

4 comments

We thank the reviewer for the constructive feedback. Below, we address the concerns with clarifications and additional quantitative and qualitative results included in the revised PDF.

Weakness 1: Missing related work

Thank you for pointing out ActAnywhere as closely related work. ActAnywhere shares similar high-level ideas for background generation and foreground harmonization with several of our baselines, including Lumen, IllumiCraft, and FlowPortal. Our intention was to focus on more recent representative methods for these related tasks. Since ActAnywhere uses an earlier Stable Diffusion architecture with AnimateDiff v2, we were concerned that a direct comparison with our Wan2.2-based model might also be influenced by differences in the pretrained backbones, making it harder to assess the compositing methods themselves. Nevertheless, we have added a citation and brief description of ActAnywhere to the related work (Sec. 2, page 3), highlighted in yellow.

Weakness 2: Foreground–target correspondence in Fig. 3

Thank you for pointing this out. The second and third foreground frames in Fig. 3 were inadvertently displayed in the wrong order relative to the corresponding target frames. We have corrected their order in Fig. 3 (page 6 of the revised PDF) and highlighted the revised figure in yellow.

Weakness 3: Unclear construction of the two sets of unaligned background references

Thank you for requesting this clarification. In our main evaluation, each of the 500 foreground videos is paired with two distinct sets of background references generated by an image generation model, yielding 1,000 cross-scene test cases. These references depict environments different from the foreground source scenes. All methods are evaluated on the same foreground–background pairs. Following the reviewer’s suggestion, we have added examples in Fig. R3 (page 10 of the revised PDF) that combine the same foreground video with five different sets of background references. These cover nighttime sodium-lamp lighting, spotlights, blue hour, golden-hour backlighting, and overcast skylight. The results show that FreeCHROMA follows the different reference environments and adapts foreground illumination accordingly, while preserving the original foreground performance.

Weakness 4: Unclear specific versions of the compared models

Thank you for these suggestions. We list the video backbone versions and scales in the table below and have also added this table to Appendix G, Table 8 (page 20 of the revised PDF), highlighted in yellow.

MethodVideo backboneBackbone scale
LumenWan2.1-Fun-1.3B-Control1.3B
VACEWan2.1-VACE-14B14B
IllumiCraftWan2.1-Fun-1.3B-Control1.3B
FlowPortalWan2.1-Fun-V1.1-1.3B-Control1.3B
ID-V2VWan2.1-I2V-14B-720P14B
BerniniWan2.2-T2V-A14B14B
FreeCHROMAWan2.2-Fun-A14B-Control14B

Weakness 5: Unclear proportion of static and large-motion cases

Following the motion-magnitude assessment in Sec. 3.2, the evaluation benchmark comprises approximately one third nearly static cases, one third with moderate camera motion, and one third with large camera motion. All methods are evaluated on the same 1,000 foreground–background pairs, and Table 3 reports results aggregated over this distribution.

Weakness 6: Unclear why RPE is not reported

RPE was omitted from Table 4 due to space constraints. We report the expanded camera-trajectory ablation, including ATE, translational and rotational RPE, and relative path length, in the table below. These results have also been added to Appendix G, Table 7 (page 20 of the revised PDF), highlighted in yellow. The full model performs best across all four metrics, further supporting the contributions of both training components to camera-motion consistency.

MethodATE ↓RPE translation ↓RPE rotation (°) ↓Relative path length → 1
Without foreground illumination perturbation0.0060.003530.129230.918
Without background viewpoint augmentation0.0050.003840.126910.909
FreeCHROMA0.0040.003180.119390.940

Weakness 7: Need clarification on methodological novelty

Thank you for raising this concern. We would like to clarify that our contribution is to efficiently adapt a pretrained video diffusion model to natively use unaligned environment images as persistent references for generative chroma keying. In creative workflows, foreground footage and environment assets are often acquired separately, so their alignment cannot be assumed. This requires training the model to infer coherent foreground–background relationships from references that do not prescribe the target viewpoint or layout, while preserving foreground fidelity.

Our supervision and data construction pipeline are specifically designed to teach this capability. Since paired examples of unaligned inputs and coherent target videos are difficult to collect naturally, we construct scalable pseudo-paired supervision in reverse from real monocular videos.

Specifically, we perturb foreground illumination and background viewpoints only on the input side while retaining the original video as a real, temporally coherent target. This reduces shortcuts based on copying foreground appearance or directly pasting backgrounds, and jointly supervises foreground preservation, reference-conditioned background generation, and harmonization.

Weakness 8: Need clarification on implementation details

Thank you for requesting these clarifications. Appendix D already provides implementation details, including the models used and the illumination-augmentation operators and sampling ranges. Following your suggestions, we have expanded Appendix D of the revised PDF to clarify clip filtering, trajectory-based keyframe selection, and automated VLM screening, with the additions highlighted in yellow.

Clip filtering. We use numerical thresholds for temporal mask stability, image sharpness, and foreground occupancy to reject clips with unstable masks, severe blur, or excessive foreground coverage.

Keyframe selection. As described in Sec. 3.2, we use ViPE-estimated camera trajectories to assess camera-motion magnitude and determine the number of source keyframes, ranging from one to three. Representative frames are selected to cover the background visible along the camera trajectory.

VLM screening. This step is fully automated. Claude Opus 4.6 receives each generated background reference together with the corresponding keyframe from the original video. The screening task asks whether the background in the reference is consistent with the background scene in the original keyframe. Candidates that change scene identity or introduce conspicuous artifacts are rejected, without manual accept/reject decisions. We will add prompt in the final version.

Question 1: two sets of unaligned background references

Please refer to response to Weakness 3.

Question 2: Distribution of static and large camera motion in evaluation benchmark

Please refer to response to Weakness 5.

Question 3: RPE omitted from Table 4

Please refer to response to Weakness 6.

Question 4: Technical novelty about augmentation

Please refer to response to Weakness 7.

Question 5: More details about data construction pipeline

Please refer to response to Weakness 8.

Question 6: More details about VLM-based filtering

Please refer to response to Weakness 8.

Question 7: Model variants in Table 1

Please refer to response to Weakness 4.

Thank you for the authors' response. I still have a few details that I would like to better understand:

  1. Were the 1K prompts used to generate the background reference images randomly generated, or were they specifically designed according to certain criteria?

  2. For the motion-magnitude assessment, what thresholds are used to separate different motion levels? A visualization of the motion-magnitude distribution would also be helpful.

  3. I understand that the implementation details are described in the paper, but I still find them difficult to reproduce from the current text. Since one of the core contributions of this work is a data-processing pipeline that enables training chroma-keying models using real-world video data, I believe the pipeline should be specified more precisely. Some parts provide exact numerical settings, e.g., Section D.1, while other parts do not report concrete thresholds or the prompts used.

  4. I am still not fully convinced about the technical novelty of FreeCHROMA's data pipeline. Using VLMs and image-editing models for data processing is not itself new; for example, Ditto: Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset adopts a conceptually similar pipeline. The component that appears relatively novel to me is the use of NormalCrafter for foreground augmentation. Overall, this makes the methodological novelty of the data pipeline seem somewhat limited.

We thank the reviewer for the follow-up questions about some details.

Follow-up Question 1: Construction of background-generation prompts

The prompts were randomly generated to cover diverse scene types and lighting conditions. We required only that the described scene could, in principle, accommodate the foreground performance, avoiding clearly implausible combinations. Beyond this basic plausibility requirement, the specific scenes and environmental elements were generated entirely at random, without additional restrictions. Each of the 500 foreground videos was paired with two distinct background-reference sets generated under these criteria, yielding 1,000 cross-scene test cases.

Follow-up Question 2: Camera-motion thresholds and distribution

We classify camera motion using rotation magnitude r and translation magnitude t, with translation normalized by median scene depth. The classification rules are applied in the order shown below.

Motion categoryCriterionReference framesApproximate proportion
Nearly staticr < 3° and t < 0.031One third
ModerateAfter excluding nearly static videos: r < 12° or t < 0.152One third
LargeOtherwise3One third

These proportions describe the 500 foreground videos. Since each foreground video is paired with two background-reference sets, the same proportions apply to the 1,000 evaluation cases.

Follow-up Question 3: Reproducibility of the data-filtering pipeline

Thank you for emphasizing reproducibility. Our pipeline comprises two stages: preliminary video filtering and pseudo-paired data construction from the retained monocular videos. The core reproducibility details for the second stage are already provided in the main paper and appendix, including the models and versions used, the construction of conditioning inputs and training targets, and the operations and parameter settings of the core augmentations.

The first stage primarily involves general video-quality checks and statistical screening, covering background sharpness, temporal continuity, hard cuts, subtitles, black borders, and foreground occupancy. We summarized these conventional screening operations without enumerating every individual threshold. Such settings primarily determine which videos are retained. Reporting them exhaustively would help reproduce the exact filtered subset, while the core procedure for constructing training examples from accepted monocular videos is already described. The motion thresholds and reference-frame counts are specified above.

Regarding model prompts, we will include the complete prompts used throughout the pipeline, including those for VLM screening and other generative-model processing steps, in the appendix of the final version.

We will also release our model checkpoints and open-source the training and inference code, together with the complete data-processing and construction code, to support reproducibility of the full workflow.

Follow-up Question 4: Novelty of the data-construction pipeline

Thank you for recognizing the practical value and overall contribution of our work. Regarding the novelty of the data-construction pipeline specifically, our contribution is a scalable solution to the lack of suitable paired supervision for generative chroma keying with unaligned environment references. This task requires supervision that jointly supports foreground preservation, reference-environment fidelity, and harmonization, yet such training pairs are difficult to collect naturally.

Our pipeline makes this supervision available from existing real monocular videos, without requiring specially captured or manually composited training pairs. We therefore emphasize the contribution at the level of the training-data formulation and its scalable construction. VLMs and image-editing models are components used to implement this formulation; the methodological contribution concerns the supervision the pipeline creates and the capability it enables. The controlled ablations further support the importance of the data-construction choices while keeping the backbone and training schedule fixed.

We hope our response has addressed your concerns. Thank you again for your time and constructive suggestions, which have helped us further improve the paper.

Comments on the paper

1 comment

We thank all reviewers for their valuable feedback and the Area Chairs for their time and effort in overseeing the review process.

Our updates are organized as follows:

  1. Manuscript revisions. We have updated the paper in response to the reviewers’ comments. All revisions are highlighted with a yellow background to make them easy to locate.

  2. Additional visual comparisons. All newly added visual comparisons are collected in Figs. R1–R3 on page 10 of the revised PDF for easy reference. These figures are also referenced in our responses to the relevant reviews.

  3. Additional quantitative results. These results are presented in tables within the corresponding reviewer responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.