acceptodds
Under review as a conference paper at ICLR 2027

TFUSE: TEMPORAL FEEDBACK GUIDES CROSS- MODAL SELECTION FOR INFRARED–VISIBLE VIDEO FUSION

Abstract

Infrared–visible video fusion requires not only high-quality fusion at the individual-frame level, but also temporal consistency across consecutive frames. However, these two objectives may conflict with each other. To address this issue, we propose TFuse. At the objective level, we introduce a learnable loss generator. The generator performs undecimated Haar wavelet decomposition on infrared and visible clips, and predicts frame-adaptive kernels for the low- and high-frequency subbands to construct dynamic intensity losses. These losses are learned through online bilevel optimization inspired by the look-ahead principle of Nesterov acceleration. A support clip performs a differentiable virtual update; a query clip evaluates the updated model using a temporal-consistency objective, and the resulting temporal feedback optimizes the loss generator. At the representation level, we introduce Temporal Enhancement with Multi-frequency Propagation Operator (TEMPO). For low-frequency structural information, it jointly models temporal dependencies using clip-level global memory and bidirectional local memory. For high-frequency details, it employs confidence-gated temporal refinement to suppress unreliable temporal propagation. During cross-modal fusion, we introduce Modality-aware Information Routing and Aggregation (MIRA), which combines the basic fused features with the weighted sum of residuals from each modality to enable adaptive information selection. Through these designs, TFuse coordinates the competing objectives, improving temporal coherence while preserving high-quality fused imagery. Code is available at the supplementary materials.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Infrared–Visible Video Fusion, Temporal Coherence, Cross-Modal Fusion, Bilevel Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PDLCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TFuse for infrared–visible video fusion. Its main idea is to resolve the conflict between frame-level fusion quality and temporal consistency by learning the fusion loss itself. A loss generator predicts frame-wise 3×3 kernels for low- and high-frequency Haar subbands, initialized from Gaussian and Laplacian priors, to form dynamic intensity losses. The generator is trained by one-step unrolled bilevel optimization: a virtual update on a support clip, followed by a temporal-consistency loss evaluated on a query clip. Two architectural modules are added. TEMPO performs frequency-aware temporal propagation, using global/local memory for low frequencies and confidence-gated adjacent-frame refinement for high frequencies. MIRA performs reference-plus-residual cross-modal routing. Experiments on M3SVD, HDO, and VTMOT show the best temporal metrics (flowD, BiSWE) and generally the best fusion metrics.

Critical strengths
  1. The motivation is interesting: rather than statically weighting fusion and temporal losses, temporal feedback is used to reshape the fusion objective. Fig. 2 provides some evidence of gradient conflict.
  2. Consistent and sizable improvements in flowD and BiSWE across three datasets.
  3. The appendix analysis of anomalous IR frames in HDO is a careful touch.
Critical weaknesses
  1. Missing key controls for the central claim. The paper argues that fixed loss weighting is insufficient, yet it never compares against the most direct alternatives: (1) joint training with L_fus + λ·L_temp under tuned λ; (2) gradient-conflict methods such as PCGrad or MGDA; and (3) the fixed prior kernels K_G and K_Δ without any learning. The last control is especially important. Gaussian pre-filtering before the max operation already smooths the selection map, which by itself may improve temporal stability. Without these controls, the benefit of bilevel learning cannot be isolated.
  2. Framing of the bilevel scheme. The procedure is a standard one-step unrolled hypergradient (as in MAML, DARTS, and Franceschi et al., 2018); the connection to Nesterov acceleration is nominal. In addition, the learnable signal is only two 3×3 kernels per frame, so it is unclear how much “cross-modal information selection” it can actually express.
  3. Unclear role of L_temp during training. According to Algorithm 1, θ is updated only with L_fus, and L_temp acts only through the loss generator. What temporal objective does each ablation variant use, in particular “+TEMPO” without bilevel learning? Please specify the full training objective for every row of Table 2.
  4. Undisclosed relationship to VideoFusion. Table 2 builds TFuse progressively from VideoFusion, and its first row matches VideoFusion’s Table 1 numbers exactly, yet Sec. 3 and Fig. 3 describe a standalone three-scale backbone. Please clarify which VideoFusion components are retained and what TEMPO and MIRA replace or add.
  5. Evaluation protocol concerns. (1) The paper states that all methods are retrained, but DDFM is training-free (it uses a pretrained DDPM), and SDMFusion relies on a pretrained diffusion prior. Please clarify how these were handled; (2) The VTMOT split (25 training / 5 test videos, randomly chosen) is very small, and the video list is not provided; and (3) The temporal training loss, which matches source inter-frame differences, is closely aligned with flowD and BiSWE. Sharpness or gradient metrics (SF, AG, Q^AB/F) should be reported to rule out stability gained through blurring.
  6. Surprising numbers. VIF on M3SVD jumps from 0.524 (VideoFusion) to 0.758, while on HDO it moves only from 0.538 to 0.540. In Table 2, MIRA alone reaches 0.642 and bilevel learning alone 0.602, but their combination reaches 0.758, which is strongly super-additive. Please explain these results.
  7. Downstream tracking is only qualitative. Fig. 7(b) shows no metrics (e.g., MOTA, IDF1, HOTA), and YOLOv11 is a detector. Please specify the tracker and report quantitative results.
  8. Other issues. (1) No comparison of parameters, FLOPs, or runtime, and no report of the training overhead of second-order bilevel updates; (2) The claim of “SOTA across all three datasets” is not supported on HDO EN and SSIM; (3) Several baselines discussed in the related work (UniVF, RCVS, USCFNet) are not compared. UniVF is a notable omission, given that the paper uses its BiSWE metric; and (4) There are numerous grammatical errors and overloaded notation (e.g., T).
Questions for authors
  1. How does TFuse compare with (a) L_fus + λ·L_temp with tuned λ, and (b) the same model using the fixed priors K_G and K_Δ without learning?
  2. What exactly is inherited from VideoFusion?
  3. What is the training objective of each variant in Table 2?
  4. How were DDFM and SDMFusion “retrained”?
  5. Can you provide quantitative tracking metrics and sharpness metrics?
34 comments
AuthorsOpenReview ↗

Response to Reviewer PDLC [Part 1/8]

Dear Reviewer PDLC,

Thank you for your thoughtful review and constructive suggestions. We appreciate your detailed questions on the design of TFuse, its optimization and ablation settings, experimental protocols, and computational costs. Below, we address each of your concerns point by point.

W1&Q1: How does TFuse compare with fixed loss weights and fixed prior kernels?

R1: Thank you for this question.

(1) For the fixed-loss-weight control experiment, we use the following fixed joint objective:

Ljoint=Lfus+λLtempL_{\mathrm{joint}} = L_{\mathrm{fus}} + \lambda L_{\mathrm{temp}}

Here, LtempL_{\mathrm{temp}} is defined in Eq. (17). The M3SVD training directory contains 190 corresponding videos, which we split into 170 training videos and 20 independent validation videos. Since full training is time-consuming, we use a fixed subset to efficiently select a suitable weight: two fixed clips of 7 consecutive frames from each training video and one fixed clip of 32 consecutive frames from each validation video. All candidate values of λ\lambda use exactly the same settings. We consider:

λ∈0.25,;1.0,;4.0\lambda \in \\{0.25,\\;1.0,\\;4.0\\}

Centered on the original, untuned value λ=1\lambda=1, these three values represent weaker, baseline, and stronger temporal supervision on a logarithmic scale. They cover a representative range of fixed weights while avoiding the selection of values to favor a particular outcome. We retrain the model for each candidate; the results are shown below.

λ\lambdaVIF ↑flowD ↓BiSWE ↓
0.250.5043.9532.470
10.5033.7762.389
40.5003.7872.453

We select the weight by considering all three metrics. The VIF at λ=0.25\lambda=0.25 is only slightly higher than at λ=1.0\lambda=1.0, while λ=1.0\lambda=1.0 gives clearly the best temporal metrics. We therefore select λ=1\lambda=1. The results above are from a low-cost screening experiment on a fixed subset, used only to select λ\lambda, and are not final test results. After selecting λ=1\lambda=1, we retrain the Fixed loss model using the same training data, number of training epochs, and evaluation protocol as full TFuse. Its results are reported in the Fixed loss row.

A fixed joint loss can only adjust the strength of the temporal constraint. An overly strong constraint may suppress genuine details along with inter-frame fluctuations. In contrast, TFuse uses bilevel optimization to convey temporal feedback to the dynamic kernels and adapt the information-selection criterion. This allows the fusion loss to preserve useful source-image structure and textures while reducing its preference for responses that introduce additional inter-frame fluctuations, thereby supporting both detail preservation and temporal consistency.

(2) We also evaluate the same model with the fixed prior kernels KGK_G and KΔK_\Delta that you suggested, without learning their parameters. The results are reported in the Fixed Kernels row. These kernels constrain the fusion loss through Gaussian low-pass filtering and Laplacian high-frequency responses, respectively. Although Gaussian prefiltering can reduce interference from local noise, the kernels act on training supervision rather than directly smoothing output frames, so their effect on temporal stability is indirect. Fixed responses also have limited ability to adapt to changes in modality responses across scenes. TFuse instead uses temporal feedback to adapt the dynamic kernels, making fusion supervision more responsive to scene changes.

ConfigurationM3SVD EN ↑M3SVD VIF ↑M3SVD MI ↑M3SVD SSIM ↑M3SVD flowD ↓M3SVD BiSWE ↓HDO EN ↑HDO VIF ↑HDO MI ↑HDO SSIM ↑HDO flowD ↓HDO BiSWE ↓
Baseline7.1130.5243.6910.6063.5592.4457.0400.5383.1530.6285.9073.911
Fixed loss7.3010.6423.9070.6202.9681.7627.2200.5393.3040.6325.1773.396
Fixed Kernels7.2910.6323.6430.5972.7361.5857.1480.4843.2220.6155.5254.027
TFuse7.3190.7583.9800.6272.3731.2067.2270.5403.3380.6354.6912.916
AuthorsOpenReview ↗

Response to Reviewer PDLC [Part 2/8]

W4&Q2: Which components does TFuse inherit from VideoFusion? Please specify which components are retained, which are replaced by TEMPO and MIRA, and what new designs TEMPO and MIRA introduce relative to VideoFusion.

R2: Thank you for this question. We clarify the relationship to VideoFusion [1] below.

(1) Retained components. TFuse retains VideoFusion's dual-modality 3D convolutional encoders, three-scale encoder–decoder structure, and basic feature reconstruction framework.

(2) Replaced components. TEMPO replaces VideoFusion's Bi-temporal Co-attention Module (BiCAM). MIRA replaces its Cross-modal Differential Reinforcement Module (CmDRM) and Complete Modality-guided Fusion Module (CMGF).

(3) New designs introduced by TEMPO and MIRA. TEMPO uses Haar wavelets to separate features into low-frequency structure and high-frequency details. For low-frequency features, it combines information from the entire clip and neighboring frames to enhance structural continuity. For high-frequency features, it selectively incorporates neighboring-frame details according to their reliability, avoiding erroneous propagation caused by motion misalignment. MIRA uses two-stage retrieval to identify reliably matched infrared and visible features as the basis for fusion, then supplements useful information unique to each modality through residual routing. This reduces incorrect fusion and the loss of complementary details. TEMPO and MIRA replace BiCAM and CmDRM/CMGF, respectively, rather than being added on top of them.

(4) Thank you for this correction. Table 2 uses VideoFusion as the baseline and progressively introduces TEMPO, MIRA, and temporal-feedback-guided bilevel loss learning to analyze the effect of each design on model performance. TEMPO and MIRA replace and improve the corresponding modules in the original network, while bilevel optimization further uses temporal feedback to guide dynamic fusion loss learning. We will expand the description of Table 2 in the revised manuscript to present the roles of these designs and their relationships more accurately.

[1] Linfeng Tang, Yeda Wang, Meiqi Gong, Zizhuo Li, Yuxin Deng, Xunpeng Yi, Chunyu Li, Han Xu, Hao Zhang, Jiayi Ma. VideoFusion: A Spatio-Temporal Collaborative Network for Multi-modal Video Fusion. CVPR, 2026.

AuthorsOpenReview ↗

Response to Reviewer PDLC [Part 3/8]

W3&Q3: What training objective is used for each ablation variant in Table 2?

R3:

(1) Thank you for this question. We distinguish temporal supervision in regular training from temporal feedback in bilevel optimization. Algorithm 1 primarily presents the latter and does not fully describe the loss components used in regular training. Configurations without bilevel optimization still use the fixed fusion loss and temporal loss from VideoFusion [1] to directly update the fusion-network parameters through joint optimization. With bilevel optimization, our temporal loss in Eq. (17) participates in regular training and also guides the dynamic fusion loss generator through hypergradients obtained from the virtual update. We will clarify this in the revised manuscript. We appreciate your keen insight and careful review.

(2) Ablation configurations in Table 2. We clarify the network architecture and training objective of each configuration below:

  • VideoFusion: The original baseline, using VideoFusion's fixed fusion loss, temporal loss, and other training constraints.

  • +TEMPO: Replaces BiCAM with TEMPO while retaining VideoFusion's training losses to evaluate TEMPO's independent contribution.

  • +TEMPO+MIRA: Builds on TEMPO and replaces CmDRM and CMGF with MIRA, retaining VideoFusion's original training losses to evaluate the architectural improvements.

  • +TEMPO+Bilevel loss learning: Retains VideoFusion's original CmDRM and CMGF and introduces bilevel optimization. A learnable dynamic fusion loss replaces the corresponding intensity term in the fixed fusion loss, and Eq. (17) is used as the temporal objective. The temporal loss participates in regular training and also provides feedback to the dynamic loss generator.

  • Full TFuse: Combines TEMPO, MIRA, and bilevel loss learning, using the dynamic fusion loss and the temporal objective in Eq. (17), and follows the bilevel optimization procedure described above.

(3) Effect of changing the temporal loss. Building on VideoFusion's inter-frame difference constraint, we introduce a cosine similarity constraint to form Eq. (17). The temporal loss thus considers both numerical discrepancies in inter-frame changes and the consistency of their directions. Since the main experiments introduce bilevel optimization together with this change to the temporal loss, we add a control experiment using TEMPO alone and replacing VideoFusion's temporal loss with Eq. (17) to distinguish their effects. The results are shown below.

ConfigurationM3SVD EN ↑M3SVD VIF ↑M3SVD MI ↑M3SVD SSIM ↑M3SVD flowD ↓M3SVD BiSWE ↓HDO EN ↑HDO VIF ↑HDO MI ↑HDO SSIM ↑HDO flowD ↓HDO BiSWE ↓
TEMPO (VideoFusion loss)7.1650.5293.7100.6113.0981.9257.0600.5353.1920.6295.4253.585
TEMPO (Eq. (17))7.1660.5293.7040.6103.0741.8397.0620.5353.1900.6295.3983.540
TEMPO + bilevel loss learning (Eq. (17))7.1750.6023.8100.6142.7981.5257.1280.5383.2390.6304.9563.194

On M3SVD, replacing only the temporal loss leaves VIF unchanged at 0.529, while flowD decreases from 3.098 to 3.074 and BiSWE from 1.925 to 1.839. Further introducing bilevel loss learning increases VIF to 0.602 and reduces flowD and BiSWE to 2.798 and 1.525, respectively. HDO exhibits a similar trend.

These results show that changing the temporal loss provides some improvement, while further introducing bilevel loss learning yields more substantial gains across the metrics. This supports the effectiveness of temporal-feedback-guided dynamic fusion loss learning.

We will add the complete training objective for each ablation configuration and expand the regular-training steps in Algorithm 1 to make the training procedure and ablation settings clearer.

Reviewer_kevvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TFuse for infrared-visible video fusion. The main idea is to learn a dynamic fusion loss using temporal feedback through online bilevel optimization. A loss generator predicts frequency-dependent kernels, and its parameters are updated according to the temporal performance of a virtually updated fusion network. The authors also introduce TEMPO for frequency-specific temporal propagation and MIRA for cross-modal feature selection. Experiments on M3SVD, HDO, and VTMOT show improvements in fusion quality and temporal consistency.

Critical strengths
  1. The motivation is reasonable. Using temporal behavior to adjust the fusion objective is interesting.
  2. The evaluation includes several recent video fusion methods and three datasets. The supplementary analysis of abnormal HDO frames is useful for checking the evaluation protocol.
Critical weaknesses
  1. Figure 2 provides an initial analysis, but the paper does not clearly show whether the learned objective reduces gradient conflict during training or how this relates to the final gains. Better temporal scores alone do not establish the proposed explanation.
  2. Equation (17) matches raw inter-frame differences of the fused video to both source modalities without motion compensation or an explicit reliability mask. Motion, occlusion, and source flicker can all affect these differences, so the reliability of this feedback needs further analysis.
  3. For downstream evaluation, Section 4.4 provides only tracking visualizations. The complete tracking pipeline is not specified beyond a pretrained YOLOv11 model, and no quantitative tracking results are reported. Besides, can the proposed method improve the object detection performance? Please provide the quantitative experiments resutls.
  4. The computational cost is not provieded. I suggest the author report parameters, FLOPs and inference time compared with other methods.
Questions for authors

Refer to Weakness.

10 comments
AuthorsOpenReview ↗

Response to Reviewer kevv [Part 1/4]

Dear Reviewer kevv,

Thank you for your thoughtful review and constructive suggestions. We appreciate your detailed questions on gradient conflict, temporal feedback, experimental validation, and computational cost. Below, we address your concerns point by point.

W1: Is gradient conflict consistently alleviated during training, and how is this related to the final performance improvements?

R1:

(1) Thank you for your insightful question, which has helped us clarify the link between gradient alignment and performance. To analyze how the learned fusion loss alleviates gradient conflict during training, we select checkpoints from models trained with either the learnable fusion loss or the fixed fusion loss at the Early (10k–25k), Middle (25k–40k), and Late (40k–50k) stages. Using the same video clips and the same set of network parameters, we compute the cosine similarity between each fusion-loss gradient and the temporal-loss gradient to compare gradient conflict across training stages.

The conflict rate (CR ↓) is the proportion of negative gradient cosine similarities, while the mean cosine similarity (Cos ↑) measures overall gradient alignment. At checkpoints from both training settings, the mean cosine similarity for the learnable fusion loss increases as training progresses, and its gradient conflict rate is generally lower than that of the fixed fusion loss. These results indicate that the learnable loss improves gradient alignment between the fusion and temporal objectives. The final performance comparison also shows improvements in fusion quality metrics and reductions in both flowD and BiSWE on M3SVD and HDO, further indicating that improved gradient alignment is consistent with the final gains in fusion quality and temporal consistency.

Stage (Learned-loss Ckpts)Fixed CR ↓Learned CR ↓Fixed Cos ↑Learned Cos ↑
Early [10k, 25k)51.39%50.19%+0.0015+0.0094
Middle [25k, 40k)43.80%39.54%+0.0076+0.0233
Late [40k, 50k]44.54%41.57%+0.0210+0.0437
Stage (Fixed-loss Ckpts)Fixed CR ↓Learned CR ↓Fixed Cos ↑Learned Cos ↑
Early [10k, 25k)52.78%51.39%-0.0186+0.0016
Middle [25k, 40k)58.24%44.54%-0.0438+0.0213
Late [40k, 50k]51.48%43.80%-0.0231+0.0247
ConfigurationM3SVD EN ↑M3SVD VIF ↑M3SVD MI ↑M3SVD SSIM ↑M3SVD flowD ↓M3SVD BiSWE ↓HDO EN ↑HDO VIF ↑HDO MI ↑HDO SSIM ↑HDO flowD ↓HDO BiSWE ↓
Fixed loss7.3010.6423.9070.6202.9681.7627.2200.5393.3040.6325.1773.396
TFuse7.3190.7583.9800.6272.3731.2067.2270.5403.3380.6354.6912.916
AuthorsOpenReview ↗

Response to Reviewer kevv [Part 2/4]

R1:

(2) Connection to performance improvements. Thank you for this question. The conflict between fusion quality and temporal consistency can impair both cross-frame stability and single-frame spatial fusion quality. In its study of nighttime infrared–visible video fusion, NVAHF [1] reports, based on experiments with more than 6200 captured images, that dynamic halation from vehicle headlights reduces illumination in dark regions of visible images, making pedestrians and road contours difficult to discern. Infrared images are unaffected by such halation and retain clearer target contours. This illustrates how dynamic illumination can change the reliability of local information in different modalities.

When local illumination changes cause visible (VI) responses to fluctuate across frames, frame-wise supervision may shift modality-selection preferences. This can cause the saliency of target contours to fluctuate between adjacent frames and weaken useful infrared (IR) contour information in the current frame. Since image fusion has no unique ground truth (GT), frame-wise fusion supervision tends to preserve the modality information with stronger responses in the current frame, whereas the temporal objective requires continuous representations of the same target. This gives rise to gradient conflict between the fusion and temporal objectives.

TFuse uses temporal feedback through bilevel optimization to adjust the dynamic loss kernels for the low- and high-frequency components, changing how different modalities contribute to the fusion-network gradient. Following Eqs. (5)–(11), a first-order Taylor expansion of the temporal loss after the virtual update gives:

\approx \mathcal{L}_{\mathrm{temp}}(\theta_s) -\eta\\, g^\top_t g_s,$$ Here, $g_t=\nabla_{\theta_s}\mathcal{L}_{\mathrm{temp}}(\theta_s)$ and $g_s=\nabla_{\theta_s}\mathcal{L}_{\mathrm{fus}}(\theta_s,\phi)$ are the gradients of the temporal and fusion losses with respect to the shared parameters, respectively. Under this first-order approximation, reducing the temporal loss after the virtual update is therefore equivalent to increasing the inner product of the two gradients, aligning their descent directions and thereby alleviating gradient conflict. This also explains the performance improvements. Temporal feedback reduces excessive supervision of information with strong single-frame responses that may cause cross-frame instability, while the spatial constraints in Eq. (16) still ensure the preservation of useful intensity, edge, and texture information. TFuse can therefore reduce interference from unreliable cross-modal responses with useful target contours, supporting both spatial fusion quality and temporal consistency. This is consistent with the improved gradient alignment observed in (1), the gains in fusion quality metrics, and the reductions in flowD and BiSWE on M3SVD and HDO. [1] Quanmin Guo, Hanlei Wang, Jianhua Yang. Night Vision Anti-Halation Method Based on Infrared and Visible Video Fusion. Sensors, 22(19): 7494, 2022.
AuthorsOpenReview ↗

Response to Reviewer kevv [Part 3/4]

W2: Reliability of temporal feedback.

R2: Thank you for raising concerns about the reliability of temporal feedback. To ensure a fair comparison, we use raw inter-frame difference matching for temporal supervision, similar to VideoFusion [2] and TemCoCo [3], rather than explicit motion compensation. The goal is to make temporal changes in the fused video consistent with those in the source videos, rather than to force adjacent fused frames to remain unchanged.

Raw inter-frame differences do not require motion compensation as a prerequisite. For example, when an object moves, both the source and fused videos exhibit corresponding inter-frame changes. If the fused video correctly preserves this motion, its inter-frame differences should follow a trend consistent with those of the source videos. Thus, genuine motion is not simply treated as a temporal error to suppress; raw frame differences themselves can provide useful information about motion. With this consideration, we focus on how optimization conflict between the fusion and temporal objectives affects video fusion, rather than explicit motion alignment or occlusion identification.

We agree with you that occlusion and source-video flicker may affect the reliability of temporal feedback. TFuse addresses this through TEMPO's confidence gating, which suppresses the propagation of unreliable high-frequency features across frames, together with spatial fusion constraints that preserve useful structural information. To further assess the effect of motion compensation, we follow the optical-flow-based compensation strategy in UniVF [4], replace raw frame differences with motion-compensated frame differences, and conduct a control experiment under the same training settings.

MethodEN ↑SSIM ↑AG ↑flowD ↓BiSWE ↓
Flow7.0300.7192.91310.1955.017
Ours7.1130.7374.78110.0195.225

The results show that TFuse clearly outperforms the optical-flow-compensated variant in fusion quality, achieving better EN, SSIM, and AG. In particular, AG reaches 4.781, substantially higher than 2.913 with optical flow compensation, while flowD is also better and the difference in BiSWE remains small. Mechanistically, TFuse's raw frame-difference supervision preserves responses to structural changes caused by genuine motion and guides dynamic fusion loss learning through bilevel optimization, allowing temporal feedback to account for both motion changes and detail preservation. Optical flow compensation places greater emphasis on residual consistency after alignment, which may weaken the constraint on the sharpness of moving edges and fail to effectively penalize blurring. Thus, without explicit motion compensation, TFuse more effectively preserves cross-modal structure and details while maintaining good temporal consistency, achieving a better balance between fusion quality and temporal stability.

[2] Linfeng Tang, Yeda Wang, Meiqi Gong, Zizhuo Li, Yuxin Deng, Xunpeng Yi, Chunyu Li, Han Xu, Hao Zhang, Jiayi Ma. VideoFusion: A Spatio-Temporal Collaborative Network for Multi-modal Video Fusion. CVPR, 2026.

[3] Meiqi Gong, Hao Zhang, Xunpeng Yi, Linfeng Tang, Jiayi Ma. TemCoCo: Temporally Consistent Multi-modal Video Fusion with Visual-Semantic Collaboration. ICCV, 2025.

[4] Zixiang Zhao, Haowen Bai, Bingxin Ke, Yukun Cui, Lilun Deng, Yulun Zhang, Kai Zhang, Konrad Schindler. A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking. NeurIPS, 2025.

Reviewer_rng9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TFuse for infrared-visible video fusion. The main idea is to use temporal feedback to learn the frame-level fusion loss instead of relying on fixed loss weights. A loss generator predicts adaptive kernels for low- and high-frequency components and is trained through a one-step bilevel update. The method also introduces TEMPO for frequency-aware temporal modeling and MIRA for cross-modal feature selection. Experiments on M3SVD, HDO, and VTMOT show improvements in fusion quality and temporal consistency, particularly on flowD and BiSWE.

Critical strengths
  1. Using temporal feedback to guide the fusion loss is an interesting idea. Instead of simply adding a temporal regularization term, the authors use temporal consistency to update the fusion objective itself. Eqs. 5–11 explain how this is achieved through a virtual update. The loss generator is only used during training, so it does not add inference overhead.

  2. The temporal consistency results are strong. On M3SVD, flowD decreases from 3.559 to 2.373 and BiSWE from 2.445 to 1.206 compared with VideoFusion. Similar improvements are reported on HDO and VTMOT. The ablation results also suggest that the proposed components contribute to the overall performance.

Critical weaknesses
  1. The contribution of bilevel feedback is not fully isolated. Fig. 2 shows a shift toward better gradient alignment with the learned fusion loss. However, the improvement appears modest for many samples, and the paper does not explain clearly how the gradients are computed or whether the fixed and learned losses are evaluated at the same network parameters. There is also no statistical analysis of the observed shift.More importantly, Table 2 lacks some key comparisons. There is no fixed fusion loss with the same temporal loss (Eq. 17), or a learned loss generator trained without bilevel feedback. Since the outer loop directly optimizes temporal consistency, and flowD/BiSWE measure related properties, it is difficult to separate the effect of bilevel learning from the training objective itself. The relationship with existing meta-learned loss and learning-to-reweight methods is also not discussed in Related Work.

  2. Some experimental results need further support. The method involves second-order optimization, cross-modal attention, and temporal memory, but training cost, inference latency, and memory usage are not reported. Training uses 7-frame clips while testing uses 25-frame clips. Given the global memory in TEMPO, it would be useful to know how performance changes with the clip length.The tracking evaluation is qualitative only, and VTMOT contains just five test videos. Quantitative tracking metrics would help here. The VIF improvement also varies substantially across datasets: 0.524 to 0.758 on M3SVD, compared with 0.538 to 0.540 on HDO. Finally, TFuse does not achieve the best EN or SSIM on HDO and VTMOT, so the claims about overall SOTA performance should be stated more carefully.

Questions for authors
  1. How are the gradient similarities in Fig. 2 computed? Are both losses evaluated at the same network parameters? Please also report the mean cosine shift with a confidence interval and the fraction of samples where the conflict increases.

  2. Can the authors compare against a fixed fusion loss with Eq. 17 and a learned loss generator without bilevel feedback? Were the existing baselines trained with comparable temporal constraints?

  3. What are the training overhead, inference latency, and memory requirements compared with VideoFusion? How does performance change with different test clip lengths?

  4. Could the authors provide quantitative tracking results on VTMOT, such as MOTA or IDF1? Also, what explains the much smaller VIF improvement on HDO compared with M3SVD?

4 comments
AuthorsOpenReview ↗

Response to Reviewer rng9 [Part 1/4]

Dear Reviewer rng9,

Thank you for your thoughtful review and constructive suggestions. We appreciate your detailed comments on gradient similarity, experimental validation, and the clarity of our presentation. Below, we address each of your concerns point by point.

W1&Q1: Gradient similarity computation.

R1: Thank you for this suggestion.

(1) Gradient similarity in Fig. 2. At the same checkpoint and on the same video, we compute the gradients of the fixed fusion loss, the learnable fusion loss, and the temporal loss with respect to the same network parameters θs\theta_s. We then compute the cosine similarity between each fusion-loss gradient and the temporal-loss gradient:

\frac{\nabla_{\theta_s}\mathcal{L}_{fus}\cdot\nabla_{\theta_s}\mathcal{L}_{temp}} {\lVert\nabla_{\theta_s}\mathcal{L}_{fus}\rVert_2\lVert\nabla_{\theta_s}\mathcal{L}_{temp}\rVert_2}$$ A negative similarity indicates gradient conflict. Using the same network parameters ensures a fair comparison. (2) **Statistical analysis.** The learnable fusion loss increases the mean gradient cosine similarity from 0.0194 under the fixed fusion loss to 0.0454, a mean change of +0.0260 (95% CI: [0.016, 0.038]). Only 16% of samples exhibit increased gradient conflict, while the proportion of samples with gradient conflict decreases from 42% to 29%. These results indicate that our method alleviates gradient conflict between the fusion and temporal objectives overall. To assess the representativeness of these results and rule out potential bias in the model weights toward the learnable loss, we conduct stage-wise evaluations at checkpoints trained with either the learnable loss or the fixed loss. The results below show a trend toward reduced gradient conflict with the learnable loss, alongside improvements in both fusion quality and temporal consistency in video fusion. | Stage (Learned-loss Ckpts) | Fixed Cos ↑ | Learned Cos ↑ | Mean ΔCos [95% CI] | Worsened Clips ↓ | | :------------------------- | ----------: | ------------: | -------------------------: | ---------------: | | Early [10k, 25k) | +0.0015 | +0.0094 | +0.0079 [+0.0040, +0.0124] | 38.0% | | Middle [25k, 40k) | +0.0076 | +0.0233 | +0.0156 [+0.0105, +0.0207] | 28.0% | | Late [40k, 50k] | +0.0210 | +0.0437 | +0.0227 [+0.0168, +0.0288] | 21.3% | | Stage (Fixed-loss Ckpts) | Fixed Cos ↑ | Learned Cos ↑ | Mean ΔCos [95% CI] | Worsened Clips ↓ | | :----------------------- | ----------: | ------------: | -------------------------: | ---------------: | | Early [10k, 25k) | -0.0186 | +0.0016 | +0.0202 [+0.0042, +0.0356] | 35.3% | | Middle [25k, 40k) | -0.0438 | +0.0213 | +0.0651 [+0.0504, +0.0788] | 21.3% | | Late [40k, 50k] | -0.0231 | +0.0247 | +0.0478 [+0.0418, +0.0543] | 16.3% | For both sources of model weights, the mean cosine similarity gains are positive, and all confidence intervals exclude zero. This indicates that the improvement does not depend on which loss was used to train the model weights. These results further validate that temporal feedback enables our method to adapt the fusion objective itself, better aligning the optimization directions of single-frame fusion quality and inter-frame temporal consistency.
AuthorsOpenReview ↗

Response to Reviewer rng9 [Part 2/4]

W1&Q2: Please add two control experiments. Do the existing baselines use comparable temporal constraints?

R2: Thank you for your expert suggestions, which have improved our experimental design and strengthened the paper. The additional control experiments are reported below.

ConfigurationM3SVD EN ↑M3SVD VIF ↑M3SVD MI ↑M3SVD SSIM ↑M3SVD flowD ↓M3SVD BiSWE ↓HDO EN ↑HDO VIF ↑HDO MI ↑HDO SSIM ↑HDO flowD ↓HDO BiSWE ↓
Fixed loss7.3010.6423.9070.6202.9681.7627.2200.5393.3040.6325.1773.396
No bilevel7.2740.6003.5290.5754.4653.7767.1990.5383.2970.6186.4755.057
Full model7.3190.7583.9800.6272.3731.2067.2270.5403.3380.6354.6912.916

(1) Control experiments. “Fixed loss” uses the fixed fusion loss together with Eq. (17). The fixed fusion loss tends to preserve stronger modality responses in each frame, whereas the temporal loss suppresses inter-frame fluctuations caused by switches in modality preference. These objectives may therefore conflict during optimization. “No bilevel (learnable loss generator)” further rules out the possibility that the improvement comes solely from the learnable loss generator itself.

(2) Comparison of temporal constraints. Thank you for this suggestion. We further examine the temporal constraints used by the baselines. Existing image fusion methods, such as U2Fusion, LRRNet, DDFM, TIMFusion, TC-MoA, and GIFNet, primarily focus on frame-level fusion quality without explicitly modeling temporal relationships across frames. Existing video fusion methods, such as VideoFusion, TemCoCo, MAVFusion, SDMFusion, and DRFusion, employ their own temporal modeling or constraint strategies, but their temporal objectives are not identical to ours. In comparison, we use L1 and cosine constraints to jointly match the magnitude and direction of inter-frame changes in the fused video to those in the source videos. We also use bilevel optimization to backpropagate temporal feedback from subsequent frames to the fusion loss generator, allowing the fusion objective to adapt to temporal performance and thereby coordinate single-frame fusion quality and inter-frame temporal consistency.

AuthorsOpenReview ↗

Response to Reviewer rng9 [Part 3/4]

W2&Q3: Computational cost and the effect of test clip length on performance.

R3: The computational costs of TFuse and VideoFusion [1] are compared below.

MethodTotal Params (M) ↓Trainable Params (M) ↓FLOPs (G/frame) ↓Training (s/step) ↓Cost (GPU-h/10k steps) ↓Inference (ms/frame) ↓Peak Training Memory (GiB/GPU) ↓
VideoFusion6.7516.751549.0080.7696.4189.637.050
TFuse7.2547.254661.7272.19518.29135.138.697

(1) Experimental setup. We use three NVIDIA RTX PRO 6000 Blackwell Max-Q GPUs with NCCL for training. Inputs consist of 7 frames at a resolution of 128 × 128, with a per-GPU batch size of 5. The inference batch size is 1.

(2) Cost analysis. Compared with VideoFusion [1], TFuse requires more training and inference time but substantially improves temporal consistency while maintaining its advantage in fusion quality, with nearly unchanged memory usage. The additional training cost mainly comes from bilevel optimization, whereas the increased inference cost mainly arises from more extensive cross-modal feature interactions and global–local temporal modeling in the wavelet domain.

(3) Test clip length. Thank you for this suggestion. We evaluate different test clip lengths on VTMOT. As the number of frames increases, EN rises slightly from 7.103 to 7.113, while MI peaks at 4.707 with 5 frames before decreasing slightly to 4.683. This indicates that longer temporal context does not substantially affect frame-level fusion quality, although the two metrics exhibit a slight trade-off. Meanwhile, flowD and BiSWE decrease from 10.506 and 5.768 to 10.019 and 5.225, respectively. Most of the improvement occurs between 3 and 7 frames, with performance largely saturating after 15 frames. These results show that TEMPO accommodates test clips longer than those used during training, improving temporal consistency while largely maintaining fusion quality.

FramesEN ↑MI ↑flowD ↓BiSWE ↓
37.1034.70110.5065.768
57.1034.70710.2025.459
77.1064.69310.0205.242
157.1134.68310.0195.226
257.1134.68310.0195.225

[1] Linfeng Tang, Yeda Wang, Meiqi Gong, Zizhuo Li, Yuxin Deng, Xunpeng Yi, Chunyu Li, Han Xu, Hao Zhang, Jiayi Ma. VideoFusion: A Spatio-Temporal Collaborative Network for Multi-modal Video Fusion. CVPR, 2026.

AuthorsOpenReview ↗

Response to Reviewer rng9 [Part 4/4]

W2&Q4: Quantitative tracking evaluation and differences in VIF improvements.

R4:

(1) Quantitative tracking experiments. Thank you for this question. The quantitative object tracking results are provided below.

MethodMOTA ↑IDF1 ↑HOTA ↑
U2Fusion50.5335.8734.76
LRRNet49.9433.3132.55
DDFM51.5734.9333.95
TIMFusion41.1931.2430.13
TCMOA50.4840.9237.87
GIFNet44.1234.2032.30
TemCoCo50.9236.6634.62
DRFusion50.3937.50‾\underline{37.50}35.15
SDMFusion51.5534.1933.39
MAVFusion49.9535.0534.18
VideoFusion51.85‾\underline{51.85}35.1834.37
Ours52.5436.6335.66‾\underline{35.66}

We evaluate object tracking on the partitioned VTMOT dataset. The five test videos cover a broad range of scenes, including daytime, nighttime, and complex motion, supporting generalizability. We use the same tracking setup as VideoFusion [1]: a tracking-by-detection framework consisting of a pretrained YOLO11s detector and a BoT-SORT tracker. Under this unified setup, we evaluate multi-object tracking on VTMOT using MOTA, IDF1, and HOTA. The results show that our method provides reliable fused video representations for downstream multi-object tracking and offers an advantage in overall tracking accuracy. We will add these implementation details to Section 4.4 of the revised manuscript and include the quantitative results in the revised supplementary material.

(2) Differences in VIF improvements. Thank you for highlighting this difference. We analyze it in terms of dataset characteristics and the fusion mechanism. Compared with M3SVD, which has clearer textures and more diverse scenes, HDO exhibits low infrared contrast, blurred visible textures, and cross-modal misalignment, as also noted in the supplementary material of VideoFusion [1]. Since TFuse's dynamic fusion loss depends on information from both modalities, these image quality issues and registration errors may affect the extraction and preservation of useful information, limiting improvements in fusion performance. Consequently, the benefits of adaptive fusion are less fully reflected in VIF gains. Other studies evaluating both M3SVD and HDO, including MAVFusion [2] and SDMFusion [3], exhibit similar patterns. Despite different training and test splits, these studies show similar trends, reflecting differences between the datasets.

[2] Xilai Li, Weijun Jiang, Xiaosong Li, Yang Liu, Hongbin Wang, Tao Ye, Huafeng Li, and Haishu Tan. MAVFusion: Efficient Infrared and Visible Video Fusion via Motion-Aware Sparse Interaction. ECCV, 2026.

[3] Jinyuan Liu, Ludan Sun, Tengyu Ma, Chunyan Yang, Zhiying Jiang, Long Ma, Risheng Liu, Xin Fan. Streaming Diffusion Model for Fast Infrared and Visible Video Fusion. CVPR, 2026.

W1&W2: Manuscript revisions.

R5:

(1) Related work. These connections to prior work are important for positioning TFuse; we appreciate your guidance. TFuse shares the bilevel optimization mechanism with meta-learned loss methods. For video tasks without fusion ground truth, we use temporal consistency feedback to learn constraints on local cross-modal responses, rather than directly learning a loss mapping. This also differs from learning-to-reweight methods that assign weights to existing losses. We will clarify these connections and differences in the related work of the revised manuscript.

(2) SOTA claims. Thank you for pointing out the wording issue. Although TFuse does not achieve the best results on every metric, it achieves the best MI, VIF, and both temporal consistency metrics on M3SVD and HDO, and also demonstrates strong overall performance on VTMOT, reflecting its combined advantages in fusion quality and temporal stability. We will revise the SOTA statements in the paper accordingly to make them more accurate and rigorous.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful evaluations, constructive suggestions, and continued discussions. We appreciate reviewer rng9's recognition of temporal-feedback-guided fusion loss learning, reviewer kevv's acknowledgment of our motivation, and reviewer PDLC's recognition of the consistent improvements in temporal consistency across three datasets.

The key innovation of TFuse is to use temporal feedback to adaptively learn the fusion objective itself. Specifically, a learnable loss generator predicts frame-adaptive kernels for low- and high-frequency subbands to construct dynamic intensity losses. Through online bilevel optimization, a differentiable virtual update is performed using the generated fusion loss, and the resulting temporal performance provides feedback to optimize the loss generator. At the representation level, Temporal Enhancement with Multi-frequency Propagation Operator (TEMPO) combines global–local dual memory with confidence-gated temporal refinement, while Modality-aware Information Routing and Aggregation (MIRA) enables adaptive cross-modal information selection. Together, these components coordinate fusion quality and temporal consistency at both the objective and representation levels.

Throughout the rebuttal, we conducted additional experiments and analyses to provide further empirical support for our contributions:

  • We conducted stage-wise gradient analyses, alignment-matched experiments, and comparisons with Fixed-loss, No-bilevel, Fixed Kernels, temporal-loss weighting, and PCGrad. We also evaluated a first-order temporal-feedback variant without explicit look-ahead to further investigate the contribution of the proposed look-ahead optimization strategy.

  • We supplemented MIRA-specific gradient analyses, ablation studies, and component interaction experiments to further investigate the effectiveness of dynamic frequency-aware supervision and adaptive cross-modal information selection.

  • We conducted controlled occlusion, source-flicker, reliability-mask, and optical-flow compensation experiments to evaluate temporal-feedback reliability. Additional quantitative comparisons across datasets further examined the balance between fusion quality, spatial-detail preservation, and temporal consistency.

  • We supplemented quantitative object detection results, extended tracking evaluations to 65 videos, and provided additional analyses of model complexity, training overhead, and inference efficiency to assess the practical applicability of TFuse.

We sincerely appreciate the reviewers' valuable feedback and constructive discussions. We believe these additional experiments and analyses further strengthen the evidence for our central contribution: leveraging temporal feedback to adaptively learn fusion supervision and guide the preservation of complementary cross-modal information, thereby improving the balance between fusion quality and temporal consistency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.