acceptodds
Withdrawn from ICLR 2027

SPDVF: Unified Infrared and Visible Video Fusion on Log-Euclidean Symmetric Positive Definite Manifold

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing infrared and visible video fusion methods treat spatial fusion and temporal modeling as separate stages in independent feature spaces. Temporal evolution is thus decoupled from fusion decisions, leaving flickering artifacts in the fused sequence. We observe that the second-order statistics of local features encode cross-modal complementarity and temporal dynamics at once, so a single descriptor can serve both roles. Such statistics take the form of symmetric positive definite (SPD) matrices, which reside on a curved manifold where linear operations are no longer valid. We propose SPDVF, a video fusion framework that describes local regions by SPD covariance descriptors and, under the Log-Euclidean metric that flattens this manifold, performs cross-modal alignment and temporal propagation in one shared tangent space without explicit motion estimation. Three components realize this design. Log-Euclidean Cross-Modal Attention aligns dual-modal second-order statistics through shifted-window cross-attention. The Basis-Kernel Bridge then converts the fused tokens into kernel mixing weights that refine Euclidean detail features. Log-Euclidean Mamba finally propagates temporal state over the same fused SPD tokens at a cost linear in clip length. Experiments on M3SVD, HDO, and VTMOT show that SPDVF improves spatial fidelity while maintaining temporal consistency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video fusion, Infrared and visible, SPD manifold

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ekDnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPDVF, a unified infrared and visible video fusion framework based on SPD manifolds, aiming to improve spatial fusion quality and temporal consistency simultaneously. Specifically, the authors represent local image regions using SPD covariance descriptors and map them into a shared Log-Euclidean tangent space. The proposed framework consists of three main components: LECA for cross-modal feature interaction, BKB for spatial detail refinement, and LEM for temporal information propagation without explicit optical flow estimation. Experiments are conducted on three benchmark datasets, M3SVD, HDO, and VTMOT, demonstrating promising performance in terms of spatial fidelity and temporal consistency.

Critical strengths
  1. The idea of unifying spatial fusion and temporal modelling within a shared Log-Euclidean SPD representation is interesting. It provides a different perspective on infrared and visible video fusion by exploiting second-order statistical information.

  2. The proposed framework integrates several complementary components to address cross-modal interaction, spatial detail preservation, and temporal consistency. The overall architecture is reasonably designed and technically sound.

  3. The authors conduct extensive experiments on three benchmark datasets and compare their method against several representative approaches. The quantitative results are promising, and the ablation experiments further demonstrate the effectiveness of the proposed components.

Critical weaknesses
  1. Although the authors explain the motivation behind the proposed framework in the Introduction, the paper lacks an intuitive illustration highlighting the differences between SPDVF and existing video fusion approaches. I suggest that the authors include a conceptual comparison figure to demonstrate the limitations of conventional methods and explain how the proposed unified representation addresses these limitations.

  2. According to Table 3, the proposed method does not demonstrate a clear advantage in computational efficiency. In particular, SPDVF requires 953.1 GFLOPs, substantially more than the competing approaches, and its inference time is also considerably longer than that of VideoFusion and TemCoCo. Given that computational efficiency is particularly important for video fusion, such a significant computational overhead raises concerns about the practicality of the proposed method.

  3. The construction of SPD covariance descriptors plays a central role in the proposed framework. However, the influence of important design choices, such as the covariance window size, descriptor dimension, and shrinkage coefficient, remains unclear. Additional sensitivity experiments would help demonstrate the robustness of the proposed representation and provide further justification for these configurations.

  4. Although Section 4.4 presents object detection results using YOLOv11, the evaluation is restricted to qualitative comparisons. It would be more convincing to provide quantitative results, such as mAP and recall, to demonstrate whether the improved fusion quality translates into better downstream performance.

Questions for authors

Please refer to the Critical Weaknesses section.

Reviewer_EVL1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SPDVF for infrared-visible video fusion. It represents local image regions with SPD covariance descriptors and uses a Log-Euclidean mapping so that cross-modal fusion and temporal modeling can operate on the same features. LECA performs cross-modal attention, BKB uses the resulting tokens to refine spatial features, and LEM applies bidirectional Mamba scanning for temporal propagation. The method is evaluated on M3SVD, HDO, and VTMOT, with improvements mainly in MI, VIF, and MS2R.

Critical strengths
  1. The shared SPD representation gives the method a clear direction. Using the same covariance-based tokens for cross-modal fusion and temporal propagation is a reasonable way to connect the two tasks. The Log-Euclidean mapping also makes it possible to apply standard attention and state-space operations to these descriptors.

  2. The spatial fusion results are strong. SPDVF obtains the best MI and VIF on all three datasets in Table 1. The paper also reports FLOPs, parameter counts, and inference time, making the computational trade-offs visible.

Critical weaknesses
  1. The temporal design and the benefit of SPD geometry need stronger evidence.

In Eqs. 8–9, LEM computes differences between neighboring tokens in a zigzag-serialized sequence. Most neighboring tokens come from different spatial locations within the same frame. At frame boundaries, they may also refer to unrelated spatial positions. The paper describes these differences as scan-transition cues, but their contribution to temporal modeling is not separately evaluated.

The ablation results also raise questions about the temporal metrics. Removing LECA or BKB actually improves BiSwE (2.2479 and 2.2403 versus 2.2712 for the full model). The authors explain that BiSwE may favor outputs with less high-frequency content. If so, BiSwE alone is not sufficient to establish better temporal consistency. In Table 1, the gap to VideoFusion on BiSwE is also fairly large on M3SVD and HDO, rather than a close second.

The benefit of SPD geometry itself is not isolated either. Table 2 compares different modules, but does not include an equivalent Euclidean representation or covariance features without the Log-Euclidean mapping. The learned attention projections in Eq. 5 also do not necessarily preserve the metric relationship established in Eq. 4. Thus, the geometric formulation is a motivation, but its practical contribution still needs to be demonstrated.

  1. The evaluation has some limitations.

SPDVF uses 953.1G FLOPs compared with 274.5G for VideoFusion, about 3.5 times as much, while inference latency increases from 141.0 ms to 237.9 ms. These costs are substantial, despite the linear complexity claimed for the temporal scanning module. A comparison with a similarly sized baseline would help put the gains in context.

Only MI and VIF are used for spatial evaluation, and each dataset has five test videos. The paper does not clearly state whether the baselines were retrained on the same splits, and no variance across runs is given. The downstream object detection results are also qualitative.

Questions for authors
  1. What is the effect of the difference augmentation in Eq. 9? Can the authors report results without it, and explain how differences across serialized frame boundaries provide useful temporal information?
  2. Can the authors compare SPDVF with an equivalent Euclidean feature model and a covariance-based model without the Log-Euclidean mapping? A parameter-matched comparison would help isolate the contribution of SPD geometry.
  3. Since removing LECA or BKB improves BiSwE, can the authors provide an additional temporal metric, such as flowD, to check whether the temporal gains hold? How do they interpret the relatively large BiSwE gap to VideoFusion on M3SVD and HDO?
  4. Were all baselines retrained on the same data splits? Can the authors report variability across runs and quantitative detection results? It would also help to compare against a baseline with similar parameter count or computational cost.
Reviewer_vEFDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for infrared and visible video fusion, termed SPDVF. The approach employs regularization to ensure that local feature covariance matrices are positive-definite, subsequently mapping them into a unified "Log-Euclidean" representation space via a matrix logarithm transformation. A "Log-Euclidean Cross-Modal Attention" mechanism is utilized to fuse the two modalities, while a "Basis-Kernel Bridge" structure—incorporating statistical tokens—is introduced to modulate fine-grained features within the Euclidean space. Furthermore, a bidirectional Mamba module is integrated to facilitate the propagation of fused tokens across video frames, eliminating the need for explicit optical flow computation. The training objective incorporates a fusion loss, a source-modality reconstruction loss, and a temporal consistency loss. Evaluations on the M3SVD, HDO, and VTMOT datasets demonstrate that the proposed method achieves state-of-the-art performance in 10 out of 12 combinations of spatial/temporal metrics and datasets. Additionally, the paper presents ablation studies, an analysis of representation decoupling, an efficiency comparison, and qualitative examples of object detection.

Critical strengths
  1. A coherent connection between statistical fusion decisions and temporal processing. Sections 3.3–3.5 specify how the covariance descriptors are regularized, vectorized, fused, and reused by the temporal module. The construction provides a concrete implementation of the proposed shared representation, while retaining a Euclidean branch for reconstruction detail. This makes the integration technically meaningful rather than merely adding a temporal loss to independent frame fusion.

  2. Encouraging results across several datasets and complementary metrics. Table 1 reports improvements in MI, VIF, and MS2R on all three datasets, with competitive rather than uniformly best BiSwE. The representation-decoupling experiment in Table 2 also supports the usefulness of coupling fusion and temporal processing.

Critical weaknesses
  1. In the primary comparisons, the incremental contribution of SPDVF over the most relevant existing methods has not been sufficiently substantiated. SMLNet, which is cited in the paper, already explores infrared and visible image fusion based on symmetric positive definite (SPD) matrices. Furthermore, MambaVF achieves flow-free video fusion through bidirectional state-space processing with linear sequence complexity; however, the discussion of related work and the main comparison tables in the paper do not include a comparison between SPDVF and MambaVF. This makes it difficult to distinguish the specific contribution of the "shared covariance representation" from existing approaches for "flow-free temporal modeling." A direct conceptual comparison, combined with experiments conducted under a unified evaluation framework, would significantly strengthen the justification for the method's innovation.

  2. The computational overhead is significant, and the analysis regarding efficiency requires further refinement. As shown in Table 3, SPDVF has 23.62M parameters, 953.1G FLOPs, and an inference latency of 237.9 ms—all significantly higher than the corresponding figures for VideoFusion (6.75M parameters, 274.5G FLOPs, and 141.0 ms latency). While SPDVF achieves improvements in fusion quality, the justification for such a significant increase in computational cost remains open to further discussion. Furthermore, although Section 4.6 mentions the parameter count of a widened version of VideoFusion, it lacks a corresponding comparison of fusion quality; consequently, it is difficult to determine whether the performance gains stem from the proposed module design or simply from an increase in model capacity.

  3. The effectiveness of the temporal mechanism lacks targeted validation. Equation (18) constrains the direction of frame differences via cosine similarity but remains insensitive to the magnitude of change; consequently, temporal variations in the same direction—even when amplified—do not incur additional penalties from this loss term. Furthermore, the handling of zero-valued frame differences in static scenes requires clarification. In contrast, the frame-boundary differencing in Equation (9) compares different spatial locations, which may yield non-zero signals even when the video is entirely static.

Questions for authors

1.Closest-method comparison. What is the conceptual and empirical difference from MambaVF under the same dataset split and evaluation protocol?

2.Quality and computational cost. What quality does the width-doubled VideoFusion mentioned in Sec. 4.6 achieve, and how do runtime and peak memory scale with clip length?

  1. Geometric attribution. With the other modules and training conditions held fixed, what changes when LECA and LEM share first-order tokens instead of Log-Euclidean covariance tokens, or use covariance tokens without the logarithm? These controls would distinguish the proposed geometry from the general benefit of sharing features.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.