acceptodds
Under review as a conference paper at ICLR 2027

Explicit Layer Modeling for Video Object Insertion and Video Layer Decomposition

Abstract

Most video editing systems still lack explicit layered video representations, limiting realistic compositing, object reuse, and consistent manipulation. This limitation is particularly evident in video object insertion and video layer decomposition, where existing methods lack direct supervision for foreground layers that capture both objects and their associated visual effects. We introduce TriLayer, a triplet video dataset containing aligned composite–background–foreground videos, where the foreground layers include both object appearance and associated visual effects. With aligned triplet supervision, TriLayer enables explicit supervised learning of layered video representations for the first time. Building on this dataset, we propose DBL-Diffusion, a dual-branch diffusion framework that jointly models scene-level RGB content and RGBA foreground layers through cross-branch interaction during denoising. We instantiate the framework in two tasks: DBL-Insert for layered object insertion, which generates explicit RGBA layers for realistic compositing and flexible post-editing, and DBL-Decompose for video layer decomposition, which recovers foreground and background layers using triplet supervision. Experiments demonstrate that explicit layer modeling substantially improves both insertion fidelity and decomposition quality.

Primary area:
generative models
Keywords:
Video Generative AI, Layered Video Representation, Video Object Insertion, Video Layer Decomposition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6XVWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to explicitly model videos with foreground and background layers, leading to two models: DBL-Insert for layered object insertion (RGB background with first frame -> RGBA foreground layers and RGB composite frames) and DBL-Decompose for video layer decomposition (RGB composite frames -> RGBA foreground layers and RGB background frames). Both models follow the same DBL-Diffusion architecture, which has a dual branch structure with LoRA for the RGB branch and DoRA for the RGBA branch joined with joint cross-attention on top of VACE-1.3B. This framework is trained with the proposed TriLayer dataset that contains composite-background-foreground triplets constructed with a data pipeline containing an object removal model and various matting models along with a guided inpainting module.

Critical strengths
  1. Layer modeling in videos is an important yet under-explored space. This paper attempts to cover fundamental tasks in this domain such as object insertion with layers, layer decomposition, and some layer-related applications based on constructed data triplets.

  2. Applications are interesting and VFX insertion and decomposition seem to be working well.

Appreciate the promise to release code, dataset and models.

Critical weaknesses
  1. One of my main concerns is the visual results. In the supplementary video, layer-based video editing results seem to have a lot of artifacts. For example, the dog in 05:14-05:21 has black holes in its body and the boundary shows flickering and dogs in 05:28-05:36 are not naturally blended without realistic shadows.

Visual results in Fig 3 are confusing since some results have enlarged crops and some don't and this is not consistent even between the same method. Also, the overlayed crops or RGBA layers in Fig 3 and Fig 4 are too big and cover the actual generated regions, so it's hard to judge the results.

Furthermore, the decomposed background results of Omnimatte Zero seem sharper with more realistic texture in the 3rd and 4th columns in Fig 4 and 1st row in Fig S27. Though the proposed method does look relatively better vs the compared methods, overall, it still seems far from being usable.

  1. The DBL-Diffusion architecture itself feels somewhat incremental with a simple LoRA-DoRA structure added on top of VACE.

[Overall] Though the problem set-up is interesting and data construction pipeline is reasonable, I feel this paper should be strengthened in terms of the method and the results to be accepted. In other words, with those parts improved in the future, I believe it would make a very nice paper.

[Not critical but worth noting] Some of the papers in references just link the arxiv version even though the paper has been accepted at a peer-reviewed venue (sometimes doesn't even have the arxiv identifier when cited as arxiv). The authors are advised to thoroughly check each reference to make sure that it is referring to the correct venue. For example:

  • "OmniInsert: Mask-free video insertion of any reference via diffusion transformer models" -> CVPR Findings 2026
  • "Wan-alpha: High-quality text-to-video generation with alpha channel" -> CVPR 2026
  • "InsertAnywhere: Geometrically Grounded and Optics-Aware Video Object Insertion" -> ECCV 2026
  • "FlowDirector: Training-Free Flow Steering for Precise Text-to-Video Editing" -> CVPR 2026
  • "UniCon: A Simple Approach to Unifying Diffusion-based Conditional Generation" -> ICLR 2025
  • "Decoupled weight decay regularization" -> ICLR 2019

(These are just to name a few from the top of the references section and there seems to be many more)

Questions for authors

Though I understand that pages are limited, it would be worth having some of the more relevant related work in the main paper.

2 comments

Appreciate the authors' response and providing results on independent benchmarks.

[W1-1] I suggest to clearly mention these limitations in the paper.

[W1-2, W1-3] The full image results in Fig S30 and S31 are helpful. I agree that OmnimatteZero has other issues like leftover shadows. However, I'm talking about the fine texture. In Fig S31, I see more obviously that the BG layer of OmnimatteZero is much sharper (water texture, grass texture). Is this a backbone issue or something else? Which VACE backbone model is DBL based on? As far as I know, OmnimatteZero uses an earlier backbone than VACE. I wonder if DBL can be trained to generate better texture.

[W2] I agree with Reviewer EyMe that a key design choice to be validated is the isolated training of DBL-Insert and DBL-Decompose and whether this has benefits over joint training (with a unified or partially unified architecture).

I understand that the rebuttal period is limited and don't expect the authors to fully train models at this point.

Thanks for adding the limitations to the paper.

My concerns are not really resolved:

  1. Backbone: Actually, VACE also has an LTX-Video-0.9 that matches to OmnimattZero's backbone, so why not use that? I think the evaluation would be more complete if the proposed method can be trained on multiple backbone architectures. Since VACE offers both Wan and LTX architectures, it makes sense to try both. I don't expect users to run this experiment at this point.
  2. Table R8: There seem to be very marginal differences between the compared variants, so not sure if any conclusions can be drawn from these results.

Additionally, regarding “joint training with a unified architecture,” if this refers to a comparison with a model simultaneously trained for insertion and decomposition, we also explored the possibility of running this experiment. ... However, the remaining rebuttal time and available GPU resources would not allow us to complete even one epoch of training for such a model, so we were unable to conduct this comparison.

: Yes, I did mean a unified model simultaneously trained for insertion and decomposition. Thanks for considering this experiment. I don't expect the authors to be able to train this model during the rebuttal. But my comment remains unresolved.

By the way these two comments seem to have the same content although the subheading below it is different: "[W2-1]. Evaluating Quality Gains in Joint RGB–RGBA Modeling with Cross-Branch Interaction. (2/3)", "[W2-1]. Evaluating Quality Gains in Joint RGB–RGBA Modeling with Cross-Branch Interaction. (3/3)". Not sure if there's supposed to be additional content or not.

Overall, though I appreciate the authors' active discussion, I don't consider my concerns to be fully resolved and plan to keep my score.

Reviewer_fASGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TriLayer, a triplet video dataset containing aligned composite videos, clean background videos, and RGBA foreground layers that include both object appearance and object-induced visual effects such as shadows and reflections. Based on this dataset, the authors propose DBL-Diffusion, a dual-branch diffusion framework with separate RGB and RGBA branches coupled through cross-attention. The framework is instantiated in two tasks: DBL-Insert for layered video object insertion and DBL-Decompose for video layer decomposition.

Critical strengths
  1. Important and well-motivated problem. Explicitly representing objects together with their associated effects is highly relevant to practical video editing. The proposed representation is more flexible than methods that directly generate a background-integrated RGB video.
  2. Potentially valuable dataset contribution. TriLayer contains 3,908 triplets covering 229 object categories and attempts to provide aligned composite, background, and effect-inclusive foreground videos. This addresses a genuine data limitation in video-layer modeling.
  3. Broad experimental coverage. The paper evaluates both insertion and decomposition, reports several perceptual and consistency metrics, and includes ablations of the dual-branch structure, LoRA–DoRA adaptation, foreground conditioning, disentangled timestep sampling, and background refinement.
Critical weaknesses

1.Limited evidence for cross-scene generalization. The evaluation benchmark is sampled from the same TriLayer distribution and contains only 65 videos. The manuscript does not sufficiently establish generalization to unseen source videos, scenes, object instances, visual effects, or data-generation pipelines. 2.Insufficient validation of semantic layer correctness. Perceptual metrics and compositing quality do not prove that shadows, reflections, transparency, and object boundaries are assigned to the correct layer. The paper lacks independent evaluation of alpha accuracy, boundary quality, temporal stability, and effect-specific separation.

Questions for authors

1.Generalization experiment: Test on unseen scenes, source videos, object instances, and effects from an independent data source. 2.Layer-quality experiment: Evaluate alpha accuracy, boundary quality, temporal consistency, compositing error, and the separation of shadows and reflections using independent annotations or controlled data.

Reviewer_EyMeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies video object insertion and layer decomposition through an explicit, effect-aware representation of foreground and background layers. It introduces TriLayer, a dataset of aligned RGB composite videos, RGB backgrounds, and RGBA foregrounds constructed using existing segmentation, matting, object-removal, and inpainting models. Building on VACE, the authors develop a dual-branch diffusion framework and train separate instances for insertion and decomposition, using branch-specific adapters to handle RGB and RGBA outputs. Experiments report improvements in object insertion and foreground reconstruction, particularly in representing effects such as shadows and reflections. Background reconstruction is less competitive with specialized reconstruction methods, and the approach incurs substantial inference cost.

Critical strengths
  1. TriLayer provides aligned composite, background, and RGBA foreground videos, including object-associated effects such as shadows and reflections. Its construction pipeline is a useful contribution if the resulting layers are sufficiently accurate.
  2. The method improves object insertion over the selected video-editing baseline and reports strong foreground reconstruction results, supporting the value of modeling foreground layers explicitly
Critical weaknesses
  1. The method does not sufficiently address the constraints that shape its design. The framework is built on VACE, a relatively expensive video-editing backbone, and relies on branch-specific adapters and other training modifications to support layered outputs with limited data. These choices are understandable, but the paper does not examine whether the main bottleneck lies in the backbone’s representation, the available supervision, or the training objective. Without that analysis, it is difficult to tell whether the added components solve the underlying difficulty or primarily accommodate the chosen backbone. A clearer diagnosis and targeted controls would make the method’s contribution more convincing.
  2. Insertion gains are often modest, background reconstruction trails specialized methods, and inference is costly. Evaluation against foreground targets produced by the same data pipeline also limits the strength of the evidence.
Questions for authors
  1. Are the predicted layers compositionally consistent? Please report whether combining the generated RGBA foreground and RGB background reproduces the intended composite video, and explain how this property is encouraged during training.
  2. How well do the decomposition results generalize beyond the data pipeline? The strong foreground results use targets produced by a similar reconstruction process. Evaluation on independently captured, rendered, or manually composited layers would help establish whether the gains reflect better decomposition.
  3. Which part of the dual-branch design is essential? A comparison between two branches with and without cross-branch attention would help distinguish the benefit of information exchange from that of simply increasing model capacity.
  4. What justifies using this design for both tasks? The paper trains separate insertion and decomposition instances, while background reconstruction trails specialized methods. What practical or empirical benefit does the common architecture provide relative to simpler task-specific models, especially given its reported inference cost?
  5. TriLayer’s foregrounds and backgrounds are produced by existing models and subsequent filtering. Calling them “ground truth” without qualification may overstate the certainty of these labels.
2 comments

I really appreciate the independent benchmark and the new single-branch ablations; they substantially strengthen the evidence for layer decomposition and for joint foreground–background prediction within that task. My central concern, however, remains unresolved. DBL-Insert and DBL-Decompose are separately trained instances, so the rebuttal does not establish an empirical benefit from using a common architecture across the two tasks. Moreover, the quality gains are not evaluated against the substantial inference cost. Please compare the proposed design with task-specific alternatives that provide the same final outputs, reporting quality together with inference time, memory, parameter count, and training cost under matched conditions. A two-branch model without cross-branch attention would also help isolate the benefit of JCA. These comparisons would clarify when the added complexity is justified. If the authors can address this concern with convincing evidence, I would be willing to raise my score to 4.

Thank you for the substantial additional experiments. The independent decomposition benchmark, compositional consistency evaluation, and controlled comparisons with dual-branch variants without JCA address several of my main concerns. In particular, the matched-parameter comparison provides convincing evidence that cross-branch interaction contributes to output quality beyond simply increasing adapter capacity. Reporting inference time and memory also makes the quality–cost trade-off much clearer.

I remain uncertain whether this expensive common architecture is preferable to lighter task-specific designs. However, I now regard that as a limitation of the work rather than a reason to discount the demonstrated value of the proposed method. The rebuttal has materially strengthened the paper, and I am raising my score from Weak Reject to Weak Accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.