- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces UAV-VIVID, a UAV-acquired infrared–visible video dataset (375 scene segments, 190,978 paired frames) comprising a registration-processed lower-altitude subset (used for all experiments), a higher-altitude subset, and an unregistered challenge subset, together with automatically generated text descriptions (BLIP-2 captions + YOLOv8 detections). It also proposes RT-FuseNet, a frame-level text-guided fusion baseline combining a differential-attention module (ARB), FiLM-style text modulation with a spatial gate (TAGM), and dual-modal PGD training augmentation. Experiments cover UAV-VIVID, M3SVD, MSRS, LLVIP, FMB and M3FD, plus downstream detection and segmentation.
Critical strengths
-
A native UAV IR–VI video dataset with scene-disjoint splits, two altitude regimes, and a raw (unregistered) subset is a potentially useful resource for the video fusion community.
-
The evaluation protocol is reasonably careful in some respects: scene-level splits, a fixed validation split for model selection, and retraining all baselines under identical source data.
-
The appendix provides useful detail on synchronization, intrinsic calibration, and privacy/release governance.
Critical weaknesses
-
Limited methodological novelty. (1) ARB is a direct application of differential attention (Ye et al., 2025) to self-attention over concatenated features, rather than a genuinely cross-modal mechanism; (2) TAGM is FiLM plus a tanh gate; (3) PGD-based training for fusion was explored in A2RNet; (4) The baseline is explicitly non-temporal, which sits awkwardly with the paper’s video framing. The appendix shows that adding temporal context degrades frame-wise quality.
-
Dataset limitations. (1) Only about half of the data (96K pairs) is evaluated; the high-altitude and unregistered subsets have no protocol or baselines; (2) Registration uses one homography per scene from keyframes at <20 m altitude, where parallax is significant, yet no quantitative cross-modal registration error is reported. The 1.2 px figure is intrinsic reprojection error, which is a different quantity; (3) Synchronization has up to 25 ms residual without a hardware trigger; (4) Text annotations are fully automatic, derived from the visible image only, and unvalidated, which is a concern for nighttime scenes; and (5) Table 1 lists the altitude as ~20–40 m, while the fusion subset is described as <20 m.
-
Result consistency concerns. In Table A1, the RT-FuseNet-1F row is identical to three decimals, including standard deviations, to the main “Ours” result in Table 2/Table 5. However, A1 states that PGD is disabled for all variants and that evaluation uses the restricted 5F-valid center-frame list, whereas the main model is PGD-trained and evaluated on the full test list. Two different models on different test lists cannot plausibly produce identical statistics. This also conflicts with Table 15, where removing PGD changes validation SSIM from 0.595 to 0.544. In addition, the 3F/5F variants report the same 6.25M parameters despite adding a temporal-attention adapter, and the 5F STD standard deviation (±0.136) is two orders of magnitude smaller than every other entry.
-
Implausible efficiency numbers. (1) RIS-Fuse is reported at 3624 GFLOPs in 13.8 ms, which exceeds the throughput of an L20; (2) VideoFusion is listed with 0.01M parameters and 0.16 GFLOPs; (3) RT-FuseNet itself takes 1332 ms per frame at 159 GFLOPs with no explanation; and (4) The BLIP-2 captioning, YOLOv8 detection, and text encoding required at inference are excluded from timing, which understates the real cost of a text-guided method.
-
Prompt ablation design. The “No-prompt” variant (SSIM 0.403) performs far worse than the plain backbone without TAGM or ARB (M1, SSIM 0.535). This contradicts the claim that TAGM reduces to identity under weak modulation, and suggests that text was simply removed from a model trained with text (a train/test mismatch). As a result, the ablation does not isolate the contribution of language guidance.
-
Evaluation choices. (1) Commonly used fusion metrics (MI, VIF, Q^AB/F, SF, AG) are absent; (2) SSIM is both a training loss and a headline metric; (3) Temporal metrics may reward smoothness: the frame-level A2RNet outperforms most video methods on BiSWE/flowD, and this is not discussed; and (4) All standard deviations are across frames, and many reported margins are small.
-
Downstream claims. (1) On LLVIP, IR-only input achieves AP50 0.9365 versus 0.8461 for RT-FuseNet; this undercuts the downstream-utility claim and is not discussed, and (2) The segmentation section mixes MSRS and FMB descriptions and uses a different baseline set for FMB (Table 13).
Questions for authors
- Please explain why the RT-FuseNet-1F results in Table A1 are identical to the main results despite different training (no PGD) and a different evaluation list. Please also report the true parameter counts for 3F/5F and check the 5F STD entry.
- How were the latencies in Table 14 measured? Please report end-to-end latency including caption generation, detection, and text encoding.
- How exactly was the No-prompt variant implemented? Was it trained without text, or was text removed at test time?
- Can you report a quantitative cross-modal registration error for the registered subset, for example on manually annotated correspondences?
- Were text-guided baselines (RIS-Fuse, UP-Fusion, AWM-Fuse) given the same text descriptions as RT-FuseNet?
Thanks to the authors for their additional explanations and clarifications. They have addressed most of my questions. Considering the value of the proposed UAV-VIVID dataset and its potential to facilitate future video fusion research, I have decided to raise my rating.