acceptodds
Under review as a conference paper at ICLR 2027

MambaVF: State Space Model for Efficient Video Fusion

Abstract

Video fusion aims to integrate complementary information from multiple source videos while preserving temporal consistency. Effective modeling of temporal dynamics is essential to this goal, yet existing methods incur substantial computational overhead from optical flow estimation and feature warping. In this paper, we present MambaVF, an efficient video fusion framework that uses state space model (SSM) to achieve temporal modeling without explicit motion estimation. First, by formulating video fusion as a sequential state update process, MambaVF captures long-range temporal dependencies with linear complexity, significantly reducing computation and memory costs. Second, the lightweight SSM-based fusion module eliminates conventional flow-guided alignment. Instead, it introduces a mutual state fusion module and a spatio-temporal bidirectional scanning mechanism to enable information aggregation across video streams. Experiments on multiple benchmarks confirm that MambaVF reaches state-of-the-art performance in different video fusion applications (multi-exposure, multi-focus, infrared-visible, medical), while reducing parameters by >90% and FLOPs by >80%, resulting in >50% shorter runtime. Code will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video fusion, image fusion, infrared-visible fusion, multi-exposure fusion, multi-focus fusion, medical video fusion

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ohUWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MambaVF, an efficient unified video fusion framework based on State Space Models (SSMs), to address the substantial computational overhead and limited efficiency of existing video fusion methods that rely on explicit optical flow estimation and feature warping. MambaVF introduces two key components: (1) a Mutual State Fusion (MuFuse) module that enables complementary information exchange between different video sources through reciprocal state interaction; and (2) a Spatio-Temporal Bidirectional (STB) scanning mechanism that combines spatial-priority and temporal-priority bidirectional scanning strategies to model spatio-temporal dependencies. These designs enable multi-frame information aggregation without explicit optical flow estimation.

The authors evaluate MambaVF on four video fusion tasks in VF-Bench, including multi-exposure fusion (MEF), multi-focus fusion (MFF), infrared-visible fusion (IVF), and medical video fusion (MVF). Experimental results demonstrate that MambaVF significantly reduces computational overhead while maintaining competitive fusion performance.

Critical strengths
  1. The paper leverages Mamba-based state space models for spatio-temporal modeling and incorporates reciprocal cross-source state interaction through MuFuse and bidirectional spatio-temporal scanning through STB. This enables video fusion without explicit optical flow estimation or feature warping, effectively reducing the computational overhead associated with conventional video fusion methods. Compared with UniVF, MambaVF reduces the number of parameters by approximately 94.3% and FLOPs by 86.6%, while achieving a 2.1× inference speedup, demonstrating its computational efficiency.

  2. The paper conducts extensive experiments on four video fusion tasks in VF-Bench, including multi-exposure, multi-focus, infrared-visible, and medical video fusion, and provides both quantitative and qualitative comparisons with various existing methods. In addition, ablation studies are conducted to evaluate the effectiveness of the key components. The experimental results demonstrate that MambaVF achieves competitive performance across different fusion tasks, indicating the generalizability and practical value of the proposed framework.

Critical weaknesses
  1. In the Introduction, the authors point out that conventional optical flow-based video fusion methods are susceptible to optical flow estimation errors in challenging scenarios involving fast motion, occlusions, and illumination changes, which may further lead to blurring, ghosting artifacts, or temporal inconsistencies in the fused results. However, the current experiments mainly focus on overall fusion performance comparisons on VF-Bench, without a systematic analysis of these challenging scenarios where optical flow estimation is prone to failure. I suggest that the authors conduct additional controlled experiments under varying motion magnitudes, occlusion levels, and illumination changes, comparing MambaVF with conventional flow-guided methods in terms of fusion quality, temporal consistency, and performance degradation. Such experiments would help verify whether the proposed flow-free modeling strategy can effectively mitigate the degradation in fusion quality caused by inaccurate optical flow estimation.

  2. Although the ablation studies demonstrate the effectiveness of the MuFuse and STB modules, the paper does not clearly establish their unique advantages over existing Mamba-based spatio-temporal modeling approaches and cross-source feature fusion designs. I suggest that the authors include additional comparative experiments with relevant existing methods to further substantiate the necessity and novelty of the proposed designs.

  3. The paper emphasizes that MambaVF achieves efficient video fusion through state space modeling and demonstrates substantial computational advantages in Table 5. However, the current experiments only report FLOPs and inference latency under a specific evaluation configuration, without analyzing computational efficiency across different input temporal window lengths and spatial resolutions. I suggest that the authors provide additional comparisons of inference latency, FLOPs, and GPU memory consumption under varying input sizes to further validate the scalability of the proposed method.

Questions for authors

Please refer to the concerns and suggestions raised in the Weaknesses section.

1 comment

Thank you for recognizing our efficient flow-free design, the extensive comparisons and ablations across four fusion tasks, and the framework's competitive performance, generalizability, and practical value.

  1. Q1: Can controlled analyses under varying motion, occlusion, and illumination substantiate the flow-free design?

    R1: We added challenging-scenario evaluations for multi-exposure, multi-focus, and infrared-visible video fusion. Within each task, we score the source videos for motion magnitude, occlusion, and low illumination, and select approximately the highest-scoring 20% under each criterion. Using the same subsets for all methods, we assess image similarity with SSIM and temporal consistency with MS2R. MambaVF achieves the highest SSIM and lowest MS2R in every evaluated subset, supporting the effectiveness of MuFuse's state interaction and complementary STB paths in aggregating spatial, temporal, and source information without explicit optical-flow alignment. Section 4.2 summarizes these findings; full details are provided in Appendix C, Table 8.

  2. Q2: What distinguishes MuFuse and STB from existing Mamba-based modeling and fusion designs?

    R2:

    • We clarified the methodological differences in Section 3.2 and Appendix B.1:

      • DAMamba adapts spatial scan orders, while GLSMamba combines global spatiotemporal scanning with local refinement; our STB uses eight fixed, complementary spatial-priority and temporal-priority paths matched across two video sources.
      • BIM serializes task features and MMMamba interleaves modality tokens, whereas our MuFuse maintains separate source recurrences and exchanges state information through cross-source projections at the outputs.
      • We provide detailed comparisons in Appendix B.1.
    • We also added encoder replacement experiments using designs adapted from GLSMamba and Vamba. MambaVF achieves the best results across all metrics in this comparison (Appendix B.2, Table 7).

    • Table 5 in Section 4.4 further shows that STB outperforms scan-count-matched variants, while removing MuFuse’s state interaction degrades fusion quality and temporal consistency.

  3. Q3: How do latency, FLOPs, and GPU memory scale with spatial resolution and temporal window length?

    R3: We added comparisons of latency, FLOPs, and peak GPU memory at spatial resolutions of 64×6464\times64, 128×128128\times128, 256×256256\times256, and 512×512512\times512 (Appendix D.2, Table 9), and at 128×128128\times128 with the internal fusion window extended to T=3,5,7,9T=3,5,7,9 (Appendix D.3, Table 10). MambaVF has the lowest FLOPs and second-lowest latency at every tested setting, remaining faster than UniVF. Its spatial FLOPs grow approximately in proportion to pixel count. For longer temporal windows, latency grows approximately in proportion to window length, while FLOPs grow more slowly than UniVF's. We discuss efficiency in Section 4.3 and provide detailed evaluation settings and comparisons in Appendix D.

Reviewer_NwKMOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a state-space-based video fusion framework designed to improve fusion quality, temporal consistency, and computational efficiency without relying on direct optical flow or feature warping. Specifically, it introduces a MuFuse module for cross-source information exchange and employs bidirectional spatiotemporal scanning to capture spatial details and temporal dependencies. Meanwhile, a hierarchical encoder and multiscale decoder are used to reconstruct the fused frames. The provided experiments demonstrate competitive performance across several video fusion tasks.

Critical strengths
  • This paper introduces a new Mamba-based framework for video fusion that eliminates explicit optical flow estimation and feature warping pipeline in UniVF, reducing sensitivity to flow errors and computational cost.

  • The mutual state fusion and spatiotemporal bidirectional scanning mechanisms are intuitive and well motivated, improving fusion performance and offering potential applicability to related vision tasks.

  • The reported results show that MambaVF improves several fusion quality metrics while substantially accelerating inference compared with the baseline UniVF.

Critical weaknesses
  • What temporal window lengths and frame-sampling strides are used during training and evaluation? Are windows processed independently, or are hidden states reused across overlapping windows?

  • Please provide further implementation details on the modifications to low-level selective-scan operator in MambaVF, including the changes to its C++ interface and CUDA forward/backward kernels described in the MuFuse Mamba Encoder section.

  • When calculating the computational redundancy of UniVF in the motivation section, which specific modules were included in the "optical flow estimation and warping" section?

  • The state-interaction ablation demonstrates the benefits of cross-source information exchange. However, further comparisons with conventional feature-fusion layers of comparable capacity are needed to validate the superiority of the proposed interaction mechanism in Eq. (4).

  • Comparing the eight-direction STB with 2D and 1D scanning changes both the scanning strategy and the number of scan paths. Comparisons with matched numbers of scans (e.g., through repeated scanning) are needed to verify whether the improvements stem from the proposed strategy rather than simply from additional scans.

  • Please include non-Mamba fusion architectures with comparable parameters and computational budgets to better clarify the use of SSMs.

Questions for authors

Please see the weaknesses section.

2 comments

Thank you for recognizing our efficient flow-free fusion, improved quality, reduced sensitivity to flow errors, and the intuitive, well-motivated MuFuse and STB designs with potential applicability to related vision tasks.

  1. Q1: What temporal windows and sampling strides are used, and are states reused across overlapping windows?

    R1: Training and quality evaluation use three consecutive frames per source, with a sampling stride of one, to reconstruct the center frame. Each window is processed independently, without hidden-state reuse across windows. Sliding-window inference also advances by one frame.

  2. Q2: What changes were made to the selective-scan C++ interface and CUDA forward/backward kernels?

    R2: The C++ interface of our selective-scan CUDA operator accepts two state-to-output projections per source and returns both self-source and cross-source contributions. The forward kernel computes each source's states once and uses them for both contributions. The backward kernel sums the two contributions to the state gradient before a single backward recurrence; each projection retains its own gradient. This preserves the computation defined by Equation (4) and numerical accuracy while reducing four state recurrences to two, with all four projections and eight scan directions retained. Section 3.2 now points to Appendix A, which details the interface, forward computation, and backward recurrence.

  3. Q3: Which UniVF modules are included in the optical flow estimation and feature warping measurements?

    R3: The category includes SEA-RAFT optical flow estimation and pixelwise warping of Transformer features using the estimated flow. Transformer feature extraction, subsequent fusion, and reconstruction are outside this category.

  4. Q4: How does the interaction in Equation (4) compare with conventional feature fusion of comparable capacity?

    R4: We added concatenation and cross-attention Transformer encoders as conventional fusion alternatives (Section 4.4, Table 4). They combine source features through concatenation or attention, while MuFuse exchanges accumulated state information through Equation (4). At comparable parameter counts, MuFuse achieves comparable overall fusion quality and better results on some metrics with fewer FLOPs. These encoder-level comparisons complement the state-interaction ablation (Table 5, Exp. III) and support the quality–efficiency balance of the overall MuFuse fusion design.

  5. Q5: Does STB benefit from its scanning strategy or simply from using more scans?

    R5: We added two variants to Section 4.4 and Table 5: Exp. I(b) repeats four spatial paths twice, and Exp. II(b) repeats two temporal paths four times. Both use eight independently parameterized branches and the same aggregation rule as STB, which uses eight distinct paths. Relative to the original scans, repeating 2D paths improves some spatial scores but worsens both temporal metrics, while repeating 1D paths also degrades SSIM, MI, and both temporal metrics. STB outperforms both variants on all quality metrics, supporting the benefit of complementary spatial and temporal ordering beyond scan count.

  6. Q6: Can non-Mamba alternatives with comparable parameter and computational budgets better justify using SSMs?

    R6: To assess the choice of encoder architecture, we compared Mamba-based MuFuse with multi-frame CNN, lightweight 3D CNN, concatenation Transformer, and cross-attention Transformer alternatives at comparable parameter counts (Section 4.4, Table 4). MambaVF outperforms both CNNs on all quality metrics and achieves comparable overall quality to the Transformers, with gains on some metrics, while requiring fewer FLOPs than every alternative. These results support choosing Mamba for its balance of fusion quality and computational efficiency.

Thanks for the authors' response. Some details are now clearer. Therefore, I tend to increase my rating.

Reviewer_paJ4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper purposes MambaVF, a video fusion method designed to make fusion faster and more lightweight without sacrificing too much quality. It evaluates the method on four tasks: multi-exposure, multi-focus, infrared-visible, and medical video fusion.

The main idea is to replace the optical flow estimation and feature warping used by conventional methods with Mamba. The model processes the two input videos separately, exchanges information between their internal states through MuFuse, and uses eight scanning directions in STB to capture spatial and temporal information. A decoder then combines the extracted features to produce the fused video frames.

Critical strengths

The paper uses a Mamba-based architecture to eliminate the need for explicit optical flow estimation and feature warping, aiming to reduce computational costs while avoiding errors introduced by inaccurate motion estimation. The paper is generally well-written and easy to follow, with a clear presentation of the previous limitations, method, and experimental results.

Critical weaknesses

Although the paper analyzes the limitations of optical-flow-based methods, it does not adequately justify why Mamba is necessary. This makes the paper feel like the authors first conducted the experiments with Mamba, then examined what strengths it offers, and only afterwards sought to incorporate those strengths into the paper's narrative.

The paper is generally easy to follow, but the figures are not very informative. As someone who is not an expert in this field, I find the results of different methods quite similar (referring to the visual fusion results, not the efficiency comparisons), and it is difficult to see where the proposed method performs better. Moreover, since the task is video fusion, the visual comparisons focus too much on individual frames, making them look more like image fusion results rather than demonstrating temporal things.

Questions for authors
  1. Why Mamba? The paper explains the limitations of optical-flow-based methods, but what makes Mamba particularly suitable for addressing these problems? How does it compare with simpler flow-free alternatives?

  2. What problem is actually solved? The efficiency improvements are clear, but is there evidence that the proposed method also handles motion-related errors better than existing approaches, particularly in challenging scenarios such as fast motion or occlusion?

  3. How significant are the improvements in fusion quality? The visual results of MambaVF and UniVF appear very similar. Could the authors clarify where the improvements are most noticeable, especially in terms of temporal consistency across video frames?

1 comment

Thank you for recognizing our efficient flow-free design, its avoidance of motion-estimation errors, and the clear presentation of prior limitations, our method, and results.

  1. Weakness 1 + Question 1: Why is Mamba suitable for video fusion, and how does it compare with simpler flow-free alternatives?

    R1:

    • We expanded Section 1 to connect video fusion's requirements to Mamba's properties. Even a short video window contains many spatial tokens: local CNNs require multiple layers to exchange distant information, while dense global attention incurs quadratic cost. Mamba aggregates context at linear cost for fixed feature and state dimensions, with input-dependent updates suited to changing exposure, focus, and modality. STB provides complementary spatial and temporal paths, and MuFuse exchanges accumulated context between sources while retaining their separate state histories. Together, they support detailed video reconstruction without explicit motion estimation or warping.

    • We also added ablation comparisons with multi-frame CNN, lightweight 3D CNN, concatenation Transformer, and cross-attention Transformer encoders at comparable parameter counts. MambaVF outperforms both CNNs on all quality metrics and achieves overall fusion quality comparable to that of the Transformer alternatives, while requiring fewer FLOPs than any of them, offering a favorable quality–efficiency balance. We report these additional comparisons in Section 4.4 and Table 4.

  2. Weakness 2 + Question 3: Where are the spatial and temporal fusion-quality improvements most visible?

    R2:

    • We expanded Section 4.2 with concrete visual comparisons. In MEF, MambaVF retains stronger cloud contrast than UniVF while preserving street visibility. In MFF, it preserves clearer grass–rock boundaries than RFL and EPT. In IVF, it retains both the road arrow's color and the infrared pedestrian silhouette; in MVF, it preserves MRI structure together with the SPECT intensity pattern.

    • For temporal evidence, we added space-time slice comparisons in Appendix E.1. Figure 9 shows clearly preserved high-frequency structures across frames, whereas some competing methods blur the transitions. Figure 10 shows that MambaVF produces smoother low-frequency brightness changes than several competing methods. We summarize these temporal improvements in Section 4.2.

  3. Question 2: Beyond efficiency, is there evidence of better fusion under fast motion or occlusion?

    R3: We added challenging-scenario evaluations across multiple video fusion tasks. Within each task, we score the source videos for motion magnitude, occlusion, and low illumination, and select approximately the highest-scoring 20% under each criterion. We assess image similarity with SSIM and temporal consistency with MS2R. MambaVF achieves the highest SSIM and lowest MS2R in every evaluated subset, supporting the effectiveness of MuFuse's state interaction and complementary STB paths in aggregating spatial, temporal, and source information without explicit optical-flow alignment. Section 4.2 summarizes these findings; full details are provided in Appendix C, Table 8.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

Dear Reviewers and Area Chairs,

We sincerely thank all reviewers and area chairs for their valuable time and thoughtful feedback.

We are pleased that the reviewers recognize the computational benefits of eliminating explicit optical flow estimation and feature warping (paJ4, NwKM, ohUW), the intuitive and well-motivated MuFuse and STB designs (NwKM), and our competitive fusion performance (paJ4, NwKM, ohUW). Reviewer ohUW also highlights the extensive evaluation, generalizability, and practical value of our framework. All three reviewers assess the presentation positively.

We have revised the manuscript and addressed the comments individually below, adding targeted experiments, implementation details, and analyses of fusion quality and efficiency. The individual responses explain the revisions and provide section references where applicable; the main additions in the revised paper are marked in blue.

Thank you again for your careful evaluation and constructive suggestions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.