- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article proposes a spatiotemporal decoupling framework D2Fusion for infrared and visible light video fusion. It addresses the issues of temporal consistency, spatial detail preservation, and cross modal alignment in video fusion. The method utilizes V-JEPA 2.0 and VideoMAE to provide pre training priors for both temporal and spatial branches. The contradiction between motion consistency and detail preservation is addressed through global temporal modulation and local spatial refinement, and results are generated through cross modal fusion and multi-scale fusion. The training objective is a spatiotemporal DPP constraint based on optical flow alignment, and a cross modal DPP constraint based on shared structure and modal specific information decomposition. The authors conducted analysis and downstream task evaluation on the M3SVD and HDO datasets.
Critical strengths
- The paper provides a clear problem definition and research on the conflicts between temporal stability, spatial details, and cross modal alignment in video fusion, and there is a good correspondence between the method design and the defined problem.
- The method utilizes V-JEPA 2.0 and VideoMAE to provide pre training priors for the temporal and spatial branches, respectively, and processes motion information and structural details through global temporal modulation and local spatial refinement, which have a certain complementarity in their effects.
- The paper employs two types of DPP constraints to handle spatiotemporal consistency and cross modal complementarity information, respectively, while considering temporal stability, spatial quality, and preservation of modal specific information within the same framework.
Critical weaknesses
- The paper does not provide a centralized explanation of the model versions, pre training weight sources, and input preprocessing methods for VFTP and VFSP. Some implementation information needs to be combined with the appendix to fully understand.
- The experiment provides overall performance results, but lacks classification statistics for different scenarios such as daytime, nighttime, rapid motion, and occlusion, and the performance differences in various scenarios are not clear enough.
Questions for authors
- Can finer grained ablation be supplemented, including separate removal of V-JEPA 2.0, VideoMAE, LGCA, spatial modulation, two DPP losses, and their quality weights?
- What are the main advantages of V-JEPA 2.0 and VideoMAE in guiding temporal and spatial branches, respectively, compared to existing video based model assisted fusion methods? What roles do the two priors play?
- Spatiotemporal DPP loss and cross modal DPP loss are used to constrain temporal consistency and modal complementarity, respectively. Can you provide a more intuitive explanation of the roles of the two types of losses?