acceptodds
Withdrawn from ICLR 2027

Decomposed and DPP-Guided Paradigm for Infrared and Visible Video Fusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Infrared and Visible Video Fusion (IVVF) integrates complementary information from heterogeneous sensors to produce informative and temporally coherent videos. However, existing methods suffer from spatio-temporal optimization imbalances, yielding motion ghosting or temporal incoherence, while inadequate cross-modal exploitation causes spatial misalignment. To address these issues, we propose Fusion, a Decomposed and Determinantal Point Process (DPP)-guided fusion network. With better prior understanding provided by large-scale visual models, Fusion enhances spatio-temporal consistency, leveraging them to modulate global motion and refine local details. Moreover, the opposing constraints between temporal consistency and spatial richness are abstracted as a quality–diversity trade-off unified under a DPP objective to balance them. Beyond intra-modal priors, we enforce a modality-invariant prior over shared geometric structures against cross-modal misalignment, preserving modality-specific cues. Extensive experiments on M3SVD establish Fusion as SOTA with strong cross-dataset generalization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Infrared and Visible Video Fusion, Spatio-temporal optimization, Determinantal Point Process, Modality-invariant

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JxFDOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes a spatiotemporal decoupling framework D2Fusion for infrared and visible light video fusion. It addresses the issues of temporal consistency, spatial detail preservation, and cross modal alignment in video fusion. The method utilizes V-JEPA 2.0 and VideoMAE to provide pre training priors for both temporal and spatial branches. The contradiction between motion consistency and detail preservation is addressed through global temporal modulation and local spatial refinement, and results are generated through cross modal fusion and multi-scale fusion. The training objective is a spatiotemporal DPP constraint based on optical flow alignment, and a cross modal DPP constraint based on shared structure and modal specific information decomposition. The authors conducted analysis and downstream task evaluation on the M3SVD and HDO datasets.

Critical strengths
  • The paper provides a clear problem definition and research on the conflicts between temporal stability, spatial details, and cross modal alignment in video fusion, and there is a good correspondence between the method design and the defined problem.
  • The method utilizes V-JEPA 2.0 and VideoMAE to provide pre training priors for the temporal and spatial branches, respectively, and processes motion information and structural details through global temporal modulation and local spatial refinement, which have a certain complementarity in their effects.
  • The paper employs two types of DPP constraints to handle spatiotemporal consistency and cross modal complementarity information, respectively, while considering temporal stability, spatial quality, and preservation of modal specific information within the same framework.
Critical weaknesses
  • The paper does not provide a centralized explanation of the model versions, pre training weight sources, and input preprocessing methods for VFTP and VFSP. Some implementation information needs to be combined with the appendix to fully understand.
  • The experiment provides overall performance results, but lacks classification statistics for different scenarios such as daytime, nighttime, rapid motion, and occlusion, and the performance differences in various scenarios are not clear enough.
Questions for authors
  • Can finer grained ablation be supplemented, including separate removal of V-JEPA 2.0, VideoMAE, LGCA, spatial modulation, two DPP losses, and their quality weights?
  • What are the main advantages of V-JEPA 2.0 and VideoMAE in guiding temporal and spatial branches, respectively, compared to existing video based model assisted fusion methods? What roles do the two priors play?
  • Spatiotemporal DPP loss and cross modal DPP loss are used to constrain temporal consistency and modal complementarity, respectively. Can you provide a more intuitive explanation of the roles of the two types of losses?
Reviewer_JNFAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents D²Fusion for infrared and visible video fusion. It uses V-JEPA 2 and VideoMAE to provide temporal and spatial priors, respectively, guiding temporal feature modulation and the generation of convolution kernels for spatial refinement. The authors introduce a DPP-based objective intended to balance temporal consistency and spatial richness. They also use optical-flow-derived cross-modal structural supervision to guide the decomposition of shared and modality-specific features, aiming to mitigate structural misalignment while preserving complementary information. Experiments on M3SVD and cross-dataset evaluation on HDO validate the effectiveness of the proposed approach.

Critical strengths

1)Using complementary pretrained video models to guide temporal modulation and spatial kernel generation is a promising design for jointly addressing temporal stability and spatial fidelity. 2)Results on M3SVD and cross-dataset evaluation on HDO provide evidence of the method’s effectiveness.

Critical weaknesses

1)Table 3 compares the proposed DPP losses against their removal, but does not compare them with optical-flow-aligned feature consistency or gradient reconstruction combined with orthogonality regularization. Consequently, the reported improvements do not establish whether the determinant-based formulation provides benefits beyond the additional supervision itself. 2)Cross-modal alignment and disentanglement lack direct validation. Structural-map reconstruction and low feature similarity do not establish geometric correction or complementary information retention; controlled misalignment tests and targeted evaluations are missing.

Questions for authors

1)How do the DPP objectives compare with optical-flow-aligned consistency and gradient reconstruction plus orthogonality regularization under matched settings? 2)Could you add controlled misalignment tests and separately ablate structural supervision and disentanglement constraints to directly evaluate alignment and complementary information retention? 3)Besides, “VideoMAE” and “V-JEPA 2.0” seem reversed in implementation details.

Reviewer_AJ9iOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Decomposed and Determinantal Point Process (DPP)-guided fusion network, termed D2Fusion. D2Fusion separately processes the temporal and spatial dimensions of multimodal video sequences and introduces a DPP-based unified spatio-temporal consistency constraint. Furthermore, this work makes a preliminary attempt to leverage a world model to construct temporal modeling priors, thereby guiding multimodal temporal modeling.

Critical strengths
  1. Decouples temporal modeling and spatial detail modeling, and introduces a unified spatio-temporal consistency constraint based on DPP.
  2. Introduces a world model into multimodal video fusion to provide temporal modeling priors.
Critical weaknesses
  1. Although this paper proposes a method for infrared-visible video fusion, only two of the ten compared methods are specifically designed for video fusion. This makes it difficult to adequately demonstrate the advantages of the proposed method over existing representative video fusion methods.
  2. Although Fig. 1 highlights the issue of cross-modal geometric misalignment, the manuscript lacks dedicated quantitative and qualitative evaluations of cross-modal geometric alignment.
  3. The proposed method is evaluated on only two datasets, M3SVD and HDO. Additional evaluations on more datasets are recommended to further validate the performance advantages and generalization capability of the proposed method.
Questions for authors
  1. Does the proposed method still maintain its performance advantages when compared with more representative existing video fusion methods?
  2. The fused image presented in Fig. 1 to illustrate "Structural Consistency" still exhibits spatial misalignment. This apparent alignment may result from the preservation of more infrared thermal information, which obscures the visible texture details and creates an illusion of structural alignment. It is recommended that the authors provide a more detailed explanation.
  3. Given that Fig. 1 highlights the cross-modal alignment capability of the proposed method, why is this capability not explicitly evaluated or analyzed in the manuscript? It is recommended that the authors provide dedicated validation of cross-modal alignment rather than relying on apparent alignment caused by the fusion process.
  4. It is recommended that the authors provide additional performance evaluations on more datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.