- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
FreeCHROMA performs generative video compositing from a masked foreground video, one to three spatially unaligned background references, and optional text. It adapts a pretrained video diffusion model through foreground-and-mask conditioning, persistent background-reference tokens, and LoRA. Its main contribution is a pseudo-pair construction pipeline that perturbs foreground illumination and background viewpoints while retaining the original video as the training target. Experiments evaluate general video quality, VLM-assessed compositing quality, estimated camera trajectories, and the effects of the training augmentations. Qualitative robotics applications are also presented.
Critical strengths
-
The supervision pipeline addresses identifiable training shortcuts. Foreground illumination perturbation discourages direct copying, while background viewpoint augmentation discourages treating reference images as aligned target frames. The ablations provide evidence that these choices improve harmonization and camera-trajectory consistency.
-
Joint composition and harmonization offer a potentially useful workflow. The model preserves the supplied foreground performance while generating its surrounding environment, reducing the need for separately prepared background sequences and subsequent appearance adjustment.
Critical weaknesses
- The necessity of a dedicated method and its methodological novelty are not sufficiently established. The architecture primarily combines established conditioning mechanisms and LoRA, with the main contribution residing in data construction. Although unaligned references offer convenience, the paper does not establish how often this constraint is a decisive bottleneck or whether it requires specialized training. A plausible alternative is to use an image-editing model, potentially coordinated by a VLM agent, to prepare an aligned and harmonized first frame or keyframes, followed by an existing compositing or propagation method such as FlowPortal where compatible. Such a pipeline may have limitations under camera motion, but a controlled comparison is needed to establish the practical advantage of learning composition directly. Without this evidence, the contribution appears relatively incremental for a specialized application.
Questions for authors
1.Contribution beyond data and fine-tuning: Under the same backbone, curated training pairs, and optimization budget, how does FreeCHROMA compare with a simple video-editing conditioning formulation? 2. Generalization from reconstruction to cross-environment composition: Training reconstructs the original video from foreground and background conditions derived from that same video, where placement, support, and occlusion relationships are already compatible. How reliably does the model generalize when a new environment requires different foreground–background relationships, such as new occlusions or contact with different support surfaces? Can you provide a targeted evaluation separating these cases from appearance harmonization? 3. Comparison with staged workflows: The paper includes ID-V2V + FLUX.2 as a first-frame-editing baseline. How are its edited frames constructed and checked for composition quality? To what extent do the observed differences arise from initial-frame preparation versus limitations of subsequent video propagation, particularly under substantial camera motion? 4. Scope beyond foreground-preserving composition: Which components of the conditioning and supervision pipeline can extend to broader video editing tasks, such as material changes, object replacement, or structural scene edits? Which depend specifically on preserving the supplied foreground performance?
3 comments
Concern 4: Comparison with staged workflows
Thank you for this suggestion. Tables 1–3 include two staged baselines, ID-V2V + Paste and ID-V2V + FLUX.2. Both prepare a composite first frame and then use ID-V2V for video propagation, following the first-frame preparation and propagation workflow discussed by the reviewer.
To further directly address the reviewer’s suggestion of multi-keyframe editing “potentially coordinated by a VLM agent,” we have added additional qualitative comparisons in Figs. R1 and R2 (in page 10) of the revised PDF. We use both GPT-Image (Agentic with GPT Astra as the VLM), a state-of-the-art closed-source agentic image-editing baseline, and FLUX.2, a state-of-the-art open-source image-editing model, to edit multiple keyframes. The edited keyframes are then propagated into videos using FlowPortal (Fig. R1) and ID-V2V (Fig. R2).
These examples illustrate a consistency challenge in multi-stage editing-and-propagation pipelines. Since the same foreground and environment can admit multiple plausible compositions, separate keyframe edits may select different background layouts. Even when individual edits appear plausible, they can provide inconsistent scene constraints for subsequent video propagation, resulting in layout drift and hallucinated transitions, including objects appearing or disappearing. The recurrence of these failures across both editing models and propagation methods highlights a system-level limitation of the tested pipelines: strong standalone image-editing performance alone does not guarantee consistent keyframe references or coherent videos.
FreeCHROMA jointly generates the video using foreground conditioning and persistent context from the original unaligned environment references. This formulation retains direct access to the original scene information throughout generation and avoids introducing conflicting constraints through intermediate keyframe edits. The comparisons illustrate its practical benefit when independently edited keyframes fail to provide consistent scene references.
Concern 5: Scope beyond foreground-preserving composition
Thank you for highlighting this broader perspective. We agree that the underlying paradigm can be extended to a wide range of tasks beyond foreground-preserving composition. More generally, a mask specifies the content to retain, references guide the generation of the remaining regions, and the resulting video is jointly harmonized. This provides a flexible formulation for tasks such as reference-guided video editing, content replacement, scene recomposition, and data augmentation. The robotics examples in Sec. 4.6 illustrate several applications of this broader paradigm. In this paper, we focus on generative chroma keying—background composition and harmonization—as a concrete and practically relevant instantiation that remains challenging even with state-of-the-art general-purpose image-editing models. Extensions to other tasks can build on the same formulation with appropriate masks, references, and task-specific training data.
We thank the authors for their response. However, given the somewhat limited novelty and questionable practical applicability of the proposed method, I maintain my original rating.
We thank the reviewer for recognizing how our generative chroma keying approach, with persistent conditioning from unaligned environment references, can support creative workflows, and for acknowledging our pipeline for deriving scalable supervision from real monocular videos. We also appreciate the recognition of our design for addressing training shortcuts, the supporting ablation evidence, and the practical value of joint composition and harmonization. Below, we address the concerns and questions point by point.
Concern 1: Methodological novelty
Thank you for raising this concern. We would like to clarify that our contribution is to efficiently adapt a pretrained I2V model to natively use unaligned environment images as persistent references for generative chroma keying. In creative workflows, foreground footage and environment assets are often acquired separately, so their alignment cannot be assumed. This requires training the model to infer coherent foreground–background relationships from references that do not prescribe the target viewpoint or layout, while preserving foreground fidelity.
Our supervision and data construction pipeline are specifically designed to teach this capability. Since paired examples of unaligned inputs and coherent target videos are difficult to collect naturally, we construct scalable pseudo-paired supervision in reverse from real monocular videos.
Concern 2: Comparison with a simple video-editing conditioning formulation
We thank the reviewer for raising this question. To assess the capabilities of state-of-the-art open-source video editing models on our task, we included VACE and Bernini as baselines in Tabs. 1–3. Both use Wan-family diffusion backbones in the 14B class, making them comparable to FreeCHROMA in backbone family and scale. This supports the fairness of the comparison in these respects, although their training data and optimization budgets are not matched.
Our pseudo-paired data are not directly compatible with the default training pipelines of VACE and Bernini. Retraining these models on our dataset would require adapting their data representations and conditioning interfaces. We therefore evaluate their released pretrained models using their default inference pipelines.
Fig. 4 illustrates foreground-preservation failures, including changes in the two characters’ appearance and identity in the middle example, and foreground scale drift in the right example. These results show that the evaluated general editing pipelines do not consistently meet the foreground-fidelity requirements of this task, motivating our task-specific conditioning and supervision.
Concern 3: Generalization from reconstruction to cross-environment composition
We thank the reviewer for raising this concern. As described in Sec. 4.1, the qualitative results in Fig. 4 and quantitative results in Tabs. 1–3 already evaluate cross-environment composition: each foreground video is paired with two different sets of target background references. The Scene Integration metrics in Tab. 2 also assess foreground–background spatial compatibility and illumination harmony within the generated scene through VLM scoring.
Learning compatible object placement, support, and occlusion relationships is a central motivation for our use of unaligned background references. As described in the main paper, these references provide environment information without prescribing a composition aligned with the supplied foreground performance. When multiple arrangements are plausible, the optional text prompt further specifies the intended spatial and interaction relationships, guiding the model toward a particular composition. During training, we retain the original real monocular video as a coherent target and construct illumination-perturbed foreground inputs and unaligned background references from it. Reconstruction thus provides supervision for learning foreground–background composition and harmonization under unaligned reference conditioning.
Occlusion and contact relationships are difficult to evaluate reliably in open-ended composition, where multiple spatial arrangements may be valid and dedicated evaluation criteria are needed. We therefore assess overall scene integration through VLM scoring, without separately reporting success rates for these relationships.