- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TFuse for infrared–visible video fusion. Its main idea is to resolve the conflict between frame-level fusion quality and temporal consistency by learning the fusion loss itself. A loss generator predicts frame-wise 3×3 kernels for low- and high-frequency Haar subbands, initialized from Gaussian and Laplacian priors, to form dynamic intensity losses. The generator is trained by one-step unrolled bilevel optimization: a virtual update on a support clip, followed by a temporal-consistency loss evaluated on a query clip. Two architectural modules are added. TEMPO performs frequency-aware temporal propagation, using global/local memory for low frequencies and confidence-gated adjacent-frame refinement for high frequencies. MIRA performs reference-plus-residual cross-modal routing. Experiments on M3SVD, HDO, and VTMOT show the best temporal metrics (flowD, BiSWE) and generally the best fusion metrics.
Critical strengths
- The motivation is interesting: rather than statically weighting fusion and temporal losses, temporal feedback is used to reshape the fusion objective. Fig. 2 provides some evidence of gradient conflict.
- Consistent and sizable improvements in flowD and BiSWE across three datasets.
- The appendix analysis of anomalous IR frames in HDO is a careful touch.
Critical weaknesses
- Missing key controls for the central claim. The paper argues that fixed loss weighting is insufficient, yet it never compares against the most direct alternatives: (1) joint training with L_fus + λ·L_temp under tuned λ; (2) gradient-conflict methods such as PCGrad or MGDA; and (3) the fixed prior kernels K_G and K_Δ without any learning. The last control is especially important. Gaussian pre-filtering before the max operation already smooths the selection map, which by itself may improve temporal stability. Without these controls, the benefit of bilevel learning cannot be isolated.
- Framing of the bilevel scheme. The procedure is a standard one-step unrolled hypergradient (as in MAML, DARTS, and Franceschi et al., 2018); the connection to Nesterov acceleration is nominal. In addition, the learnable signal is only two 3×3 kernels per frame, so it is unclear how much “cross-modal information selection” it can actually express.
- Unclear role of L_temp during training. According to Algorithm 1, θ is updated only with L_fus, and L_temp acts only through the loss generator. What temporal objective does each ablation variant use, in particular “+TEMPO” without bilevel learning? Please specify the full training objective for every row of Table 2.
- Undisclosed relationship to VideoFusion. Table 2 builds TFuse progressively from VideoFusion, and its first row matches VideoFusion’s Table 1 numbers exactly, yet Sec. 3 and Fig. 3 describe a standalone three-scale backbone. Please clarify which VideoFusion components are retained and what TEMPO and MIRA replace or add.
- Evaluation protocol concerns. (1) The paper states that all methods are retrained, but DDFM is training-free (it uses a pretrained DDPM), and SDMFusion relies on a pretrained diffusion prior. Please clarify how these were handled; (2) The VTMOT split (25 training / 5 test videos, randomly chosen) is very small, and the video list is not provided; and (3) The temporal training loss, which matches source inter-frame differences, is closely aligned with flowD and BiSWE. Sharpness or gradient metrics (SF, AG, Q^AB/F) should be reported to rule out stability gained through blurring.
- Surprising numbers. VIF on M3SVD jumps from 0.524 (VideoFusion) to 0.758, while on HDO it moves only from 0.538 to 0.540. In Table 2, MIRA alone reaches 0.642 and bilevel learning alone 0.602, but their combination reaches 0.758, which is strongly super-additive. Please explain these results.
- Downstream tracking is only qualitative. Fig. 7(b) shows no metrics (e.g., MOTA, IDF1, HOTA), and YOLOv11 is a detector. Please specify the tracker and report quantitative results.
- Other issues. (1) No comparison of parameters, FLOPs, or runtime, and no report of the training overhead of second-order bilevel updates; (2) The claim of “SOTA across all three datasets” is not supported on HDO EN and SSIM; (3) Several baselines discussed in the related work (UniVF, RCVS, USCFNet) are not compared. UniVF is a notable omission, given that the paper uses its BiSWE metric; and (4) There are numerous grammatical errors and overloaded notation (e.g., T).
Questions for authors
- How does TFuse compare with (a) L_fus + λ·L_temp with tuned λ, and (b) the same model using the fixed priors K_G and K_Δ without learning?
- What exactly is inherited from VideoFusion?
- What is the training objective of each variant in Table 2?
- How were DDFM and SDMFusion “retrained”?
- Can you provide quantitative tracking metrics and sharpness metrics?
34 comments
Response to Reviewer PDLC [Part 2/8]
W4&Q2: Which components does TFuse inherit from VideoFusion? Please specify which components are retained, which are replaced by TEMPO and MIRA, and what new designs TEMPO and MIRA introduce relative to VideoFusion.
R2: Thank you for this question. We clarify the relationship to VideoFusion [1] below.
(1) Retained components. TFuse retains VideoFusion's dual-modality 3D convolutional encoders, three-scale encoder–decoder structure, and basic feature reconstruction framework.
(2) Replaced components. TEMPO replaces VideoFusion's Bi-temporal Co-attention Module (BiCAM). MIRA replaces its Cross-modal Differential Reinforcement Module (CmDRM) and Complete Modality-guided Fusion Module (CMGF).
(3) New designs introduced by TEMPO and MIRA. TEMPO uses Haar wavelets to separate features into low-frequency structure and high-frequency details. For low-frequency features, it combines information from the entire clip and neighboring frames to enhance structural continuity. For high-frequency features, it selectively incorporates neighboring-frame details according to their reliability, avoiding erroneous propagation caused by motion misalignment. MIRA uses two-stage retrieval to identify reliably matched infrared and visible features as the basis for fusion, then supplements useful information unique to each modality through residual routing. This reduces incorrect fusion and the loss of complementary details. TEMPO and MIRA replace BiCAM and CmDRM/CMGF, respectively, rather than being added on top of them.
(4) Thank you for this correction. Table 2 uses VideoFusion as the baseline and progressively introduces TEMPO, MIRA, and temporal-feedback-guided bilevel loss learning to analyze the effect of each design on model performance. TEMPO and MIRA replace and improve the corresponding modules in the original network, while bilevel optimization further uses temporal feedback to guide dynamic fusion loss learning. We will expand the description of Table 2 in the revised manuscript to present the roles of these designs and their relationships more accurately.
[1] Linfeng Tang, Yeda Wang, Meiqi Gong, Zizhuo Li, Yuxin Deng, Xunpeng Yi, Chunyu Li, Han Xu, Hao Zhang, Jiayi Ma. VideoFusion: A Spatio-Temporal Collaborative Network for Multi-modal Video Fusion. CVPR, 2026.
Response to Reviewer PDLC [Part 3/8]
W3&Q3: What training objective is used for each ablation variant in Table 2?
R3:
(1) Thank you for this question. We distinguish temporal supervision in regular training from temporal feedback in bilevel optimization. Algorithm 1 primarily presents the latter and does not fully describe the loss components used in regular training. Configurations without bilevel optimization still use the fixed fusion loss and temporal loss from VideoFusion [1] to directly update the fusion-network parameters through joint optimization. With bilevel optimization, our temporal loss in Eq. (17) participates in regular training and also guides the dynamic fusion loss generator through hypergradients obtained from the virtual update. We will clarify this in the revised manuscript. We appreciate your keen insight and careful review.
(2) Ablation configurations in Table 2. We clarify the network architecture and training objective of each configuration below:
-
VideoFusion: The original baseline, using VideoFusion's fixed fusion loss, temporal loss, and other training constraints.
-
+TEMPO: Replaces BiCAM with TEMPO while retaining VideoFusion's training losses to evaluate TEMPO's independent contribution.
-
+TEMPO+MIRA: Builds on TEMPO and replaces CmDRM and CMGF with MIRA, retaining VideoFusion's original training losses to evaluate the architectural improvements.
-
+TEMPO+Bilevel loss learning: Retains VideoFusion's original CmDRM and CMGF and introduces bilevel optimization. A learnable dynamic fusion loss replaces the corresponding intensity term in the fixed fusion loss, and Eq. (17) is used as the temporal objective. The temporal loss participates in regular training and also provides feedback to the dynamic loss generator.
-
Full TFuse: Combines TEMPO, MIRA, and bilevel loss learning, using the dynamic fusion loss and the temporal objective in Eq. (17), and follows the bilevel optimization procedure described above.
(3) Effect of changing the temporal loss. Building on VideoFusion's inter-frame difference constraint, we introduce a cosine similarity constraint to form Eq. (17). The temporal loss thus considers both numerical discrepancies in inter-frame changes and the consistency of their directions. Since the main experiments introduce bilevel optimization together with this change to the temporal loss, we add a control experiment using TEMPO alone and replacing VideoFusion's temporal loss with Eq. (17) to distinguish their effects. The results are shown below.
| Configuration | M3SVD EN ↑ | M3SVD VIF ↑ | M3SVD MI ↑ | M3SVD SSIM ↑ | M3SVD flowD ↓ | M3SVD BiSWE ↓ | HDO EN ↑ | HDO VIF ↑ | HDO MI ↑ | HDO SSIM ↑ | HDO flowD ↓ | HDO BiSWE ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TEMPO (VideoFusion loss) | 7.165 | 0.529 | 3.710 | 0.611 | 3.098 | 1.925 | 7.060 | 0.535 | 3.192 | 0.629 | 5.425 | 3.585 |
| TEMPO (Eq. (17)) | 7.166 | 0.529 | 3.704 | 0.610 | 3.074 | 1.839 | 7.062 | 0.535 | 3.190 | 0.629 | 5.398 | 3.540 |
| TEMPO + bilevel loss learning (Eq. (17)) | 7.175 | 0.602 | 3.810 | 0.614 | 2.798 | 1.525 | 7.128 | 0.538 | 3.239 | 0.630 | 4.956 | 3.194 |
On M3SVD, replacing only the temporal loss leaves VIF unchanged at 0.529, while flowD decreases from 3.098 to 3.074 and BiSWE from 1.925 to 1.839. Further introducing bilevel loss learning increases VIF to 0.602 and reduces flowD and BiSWE to 2.798 and 1.525, respectively. HDO exhibits a similar trend.
These results show that changing the temporal loss provides some improvement, while further introducing bilevel loss learning yields more substantial gains across the metrics. This supports the effectiveness of temporal-feedback-guided dynamic fusion loss learning.
We will add the complete training objective for each ablation configuration and expand the regular-training steps in Algorithm 1 to make the training procedure and ablation settings clearer.
Response to Reviewer PDLC [Part 1/8]
Dear Reviewer PDLC,
Thank you for your thoughtful review and constructive suggestions. We appreciate your detailed questions on the design of TFuse, its optimization and ablation settings, experimental protocols, and computational costs. Below, we address each of your concerns point by point.
R1: Thank you for this question.
(1) For the fixed-loss-weight control experiment, we use the following fixed joint objective:
Ljoint=Lfus+λLtemp
Here, Ltemp is defined in Eq. (17). The M3SVD training directory contains 190 corresponding videos, which we split into 170 training videos and 20 independent validation videos. Since full training is time-consuming, we use a fixed subset to efficiently select a suitable weight: two fixed clips of 7 consecutive frames from each training video and one fixed clip of 32 consecutive frames from each validation video. All candidate values of λ use exactly the same settings. We consider:
λ∈0.25,;1.0,;4.0
Centered on the original, untuned value λ=1, these three values represent weaker, baseline, and stronger temporal supervision on a logarithmic scale. They cover a representative range of fixed weights while avoiding the selection of values to favor a particular outcome. We retrain the model for each candidate; the results are shown below.
We select the weight by considering all three metrics. The VIF at λ=0.25 is only slightly higher than at λ=1.0, while λ=1.0 gives clearly the best temporal metrics. We therefore select λ=1. The results above are from a low-cost screening experiment on a fixed subset, used only to select λ, and are not final test results. After selecting λ=1, we retrain the Fixed loss model using the same training data, number of training epochs, and evaluation protocol as full TFuse. Its results are reported in the Fixed loss row.
A fixed joint loss can only adjust the strength of the temporal constraint. An overly strong constraint may suppress genuine details along with inter-frame fluctuations. In contrast, TFuse uses bilevel optimization to convey temporal feedback to the dynamic kernels and adapt the information-selection criterion. This allows the fusion loss to preserve useful source-image structure and textures while reducing its preference for responses that introduce additional inter-frame fluctuations, thereby supporting both detail preservation and temporal consistency.
(2) We also evaluate the same model with the fixed prior kernels KG and KΔ that you suggested, without learning their parameters. The results are reported in the Fixed Kernels row. These kernels constrain the fusion loss through Gaussian low-pass filtering and Laplacian high-frequency responses, respectively. Although Gaussian prefiltering can reduce interference from local noise, the kernels act on training supervision rather than directly smoothing output frames, so their effect on temporal stability is indirect. Fixed responses also have limited ability to adapt to changes in modality responses across scenes. TFuse instead uses temporal feedback to adapt the dynamic kernels, making fusion supervision more responsive to scene changes.