- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes to explicitly model videos with foreground and background layers, leading to two models: DBL-Insert for layered object insertion (RGB background with first frame -> RGBA foreground layers and RGB composite frames) and DBL-Decompose for video layer decomposition (RGB composite frames -> RGBA foreground layers and RGB background frames). Both models follow the same DBL-Diffusion architecture, which has a dual branch structure with LoRA for the RGB branch and DoRA for the RGBA branch joined with joint cross-attention on top of VACE-1.3B. This framework is trained with the proposed TriLayer dataset that contains composite-background-foreground triplets constructed with a data pipeline containing an object removal model and various matting models along with a guided inpainting module.
Critical strengths
-
Layer modeling in videos is an important yet under-explored space. This paper attempts to cover fundamental tasks in this domain such as object insertion with layers, layer decomposition, and some layer-related applications based on constructed data triplets.
-
Applications are interesting and VFX insertion and decomposition seem to be working well.
Appreciate the promise to release code, dataset and models.
Critical weaknesses
- One of my main concerns is the visual results. In the supplementary video, layer-based video editing results seem to have a lot of artifacts. For example, the dog in 05:14-05:21 has black holes in its body and the boundary shows flickering and dogs in 05:28-05:36 are not naturally blended without realistic shadows.
Visual results in Fig 3 are confusing since some results have enlarged crops and some don't and this is not consistent even between the same method. Also, the overlayed crops or RGBA layers in Fig 3 and Fig 4 are too big and cover the actual generated regions, so it's hard to judge the results.
Furthermore, the decomposed background results of Omnimatte Zero seem sharper with more realistic texture in the 3rd and 4th columns in Fig 4 and 1st row in Fig S27. Though the proposed method does look relatively better vs the compared methods, overall, it still seems far from being usable.
- The DBL-Diffusion architecture itself feels somewhat incremental with a simple LoRA-DoRA structure added on top of VACE.
[Overall] Though the problem set-up is interesting and data construction pipeline is reasonable, I feel this paper should be strengthened in terms of the method and the results to be accepted. In other words, with those parts improved in the future, I believe it would make a very nice paper.
[Not critical but worth noting] Some of the papers in references just link the arxiv version even though the paper has been accepted at a peer-reviewed venue (sometimes doesn't even have the arxiv identifier when cited as arxiv). The authors are advised to thoroughly check each reference to make sure that it is referring to the correct venue. For example:
- "OmniInsert: Mask-free video insertion of any reference via diffusion transformer models" -> CVPR Findings 2026
- "Wan-alpha: High-quality text-to-video generation with alpha channel" -> CVPR 2026
- "InsertAnywhere: Geometrically Grounded and Optics-Aware Video Object Insertion" -> ECCV 2026
- "FlowDirector: Training-Free Flow Steering for Precise Text-to-Video Editing" -> CVPR 2026
- "UniCon: A Simple Approach to Unifying Diffusion-based Conditional Generation" -> ICLR 2025
- "Decoupled weight decay regularization" -> ICLR 2019
(These are just to name a few from the top of the references section and there seems to be many more)
Questions for authors
Though I understand that pages are limited, it would be worth having some of the more relevant related work in the main paper.
2 comments
Thanks for adding the limitations to the paper.
My concerns are not really resolved:
- Backbone: Actually, VACE also has an LTX-Video-0.9 that matches to OmnimattZero's backbone, so why not use that? I think the evaluation would be more complete if the proposed method can be trained on multiple backbone architectures. Since VACE offers both Wan and LTX architectures, it makes sense to try both. I don't expect users to run this experiment at this point.
- Table R8: There seem to be very marginal differences between the compared variants, so not sure if any conclusions can be drawn from these results.
Additionally, regarding “joint training with a unified architecture,” if this refers to a comparison with a model simultaneously trained for insertion and decomposition, we also explored the possibility of running this experiment. ... However, the remaining rebuttal time and available GPU resources would not allow us to complete even one epoch of training for such a model, so we were unable to conduct this comparison.
: Yes, I did mean a unified model simultaneously trained for insertion and decomposition. Thanks for considering this experiment. I don't expect the authors to be able to train this model during the rebuttal. But my comment remains unresolved.
By the way these two comments seem to have the same content although the subheading below it is different: "[W2-1]. Evaluating Quality Gains in Joint RGB–RGBA Modeling with Cross-Branch Interaction. (2/3)", "[W2-1]. Evaluating Quality Gains in Joint RGB–RGBA Modeling with Cross-Branch Interaction. (3/3)". Not sure if there's supposed to be additional content or not.
Overall, though I appreciate the authors' active discussion, I don't consider my concerns to be fully resolved and plan to keep my score.
Appreciate the authors' response and providing results on independent benchmarks.
[W1-1] I suggest to clearly mention these limitations in the paper.
[W1-2, W1-3] The full image results in Fig S30 and S31 are helpful. I agree that OmnimatteZero has other issues like leftover shadows. However, I'm talking about the fine texture. In Fig S31, I see more obviously that the BG layer of OmnimatteZero is much sharper (water texture, grass texture). Is this a backbone issue or something else? Which VACE backbone model is DBL based on? As far as I know, OmnimatteZero uses an earlier backbone than VACE. I wonder if DBL can be trained to generate better texture.
[W2] I agree with Reviewer EyMe that a key design choice to be validated is the isolated training of DBL-Insert and DBL-Decompose and whether this has benefits over joint training (with a unified or partially unified architecture).
I understand that the rebuttal period is limited and don't expect the authors to fully train models at this point.