acceptodds
Under review as a conference paper at ICLR 2027

UAV-VIVID: A Large-Scale UAV Infrared–Visible Video Dataset and a Text-Guided Frame-Level Fusion Baseline

Abstract

Infrared–visible (IR–VI) fusion is commonly developed on independent, well-aligned image pairs, leaving its behavior on continuous UAV streams insufficiently characterized. We introduce UAV-VIVID, a UAV-acquired IR–VI video dataset containing 375 temporally continuous scene segments and 190,978 paired frames. Its 96,087 registration-processed lower-altitude pairs define the fusion track evaluated in this work, while 79,992 higher-altitude pairs and 14,899 unregistered lower-altitude pairs are released as auxiliary resources. The dataset preserves temporal order, adopts scene-disjoint splits, and provides structured textual descriptions. To establish a controlled frame-level reference for this video setting, we further present the Residual Text-Guided Fusion Network (RT-FuseNet), which processes each IR–VI pair independently without neighboring frames or temporal state. Adaptive Residual Balancing regulates cross-modal interactions, while Text-Affine and Gated Modulation injects language guidance through channel-wise affine parameters and spatial gates. Experiments across multiple datasets evaluate frame-wise fusion quality, sequence-level output behavior, text conditioning, downstream utility, and the clean-input regularization effect of training-time dual-modal PGD augmentation.

Primary area:
datasets and benchmarks
Keywords:
Infrared--visible fusion, UAV infrared--visible video dataset, multimodal fusion benchmark, temporal consistency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zoi3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces UAV-VIVID, a UAV-acquired infrared–visible video dataset (375 scene segments, 190,978 paired frames) comprising a registration-processed lower-altitude subset (used for all experiments), a higher-altitude subset, and an unregistered challenge subset, together with automatically generated text descriptions (BLIP-2 captions + YOLOv8 detections). It also proposes RT-FuseNet, a frame-level text-guided fusion baseline combining a differential-attention module (ARB), FiLM-style text modulation with a spatial gate (TAGM), and dual-modal PGD training augmentation. Experiments cover UAV-VIVID, M3SVD, MSRS, LLVIP, FMB and M3FD, plus downstream detection and segmentation.

Critical strengths
  1. A native UAV IR–VI video dataset with scene-disjoint splits, two altitude regimes, and a raw (unregistered) subset is a potentially useful resource for the video fusion community.

  2. The evaluation protocol is reasonably careful in some respects: scene-level splits, a fixed validation split for model selection, and retraining all baselines under identical source data.

  3. The appendix provides useful detail on synchronization, intrinsic calibration, and privacy/release governance.

Critical weaknesses
  1. Limited methodological novelty. (1) ARB is a direct application of differential attention (Ye et al., 2025) to self-attention over concatenated features, rather than a genuinely cross-modal mechanism; (2) TAGM is FiLM plus a tanh gate; (3) PGD-based training for fusion was explored in A2RNet; (4) The baseline is explicitly non-temporal, which sits awkwardly with the paper’s video framing. The appendix shows that adding temporal context degrades frame-wise quality.

  2. Dataset limitations. (1) Only about half of the data (96K pairs) is evaluated; the high-altitude and unregistered subsets have no protocol or baselines; (2) Registration uses one homography per scene from keyframes at <20 m altitude, where parallax is significant, yet no quantitative cross-modal registration error is reported. The 1.2 px figure is intrinsic reprojection error, which is a different quantity; (3) Synchronization has up to 25 ms residual without a hardware trigger; (4) Text annotations are fully automatic, derived from the visible image only, and unvalidated, which is a concern for nighttime scenes; and (5) Table 1 lists the altitude as ~20–40 m, while the fusion subset is described as <20 m.

  3. Result consistency concerns. In Table A1, the RT-FuseNet-1F row is identical to three decimals, including standard deviations, to the main “Ours” result in Table 2/Table 5. However, A1 states that PGD is disabled for all variants and that evaluation uses the restricted 5F-valid center-frame list, whereas the main model is PGD-trained and evaluated on the full test list. Two different models on different test lists cannot plausibly produce identical statistics. This also conflicts with Table 15, where removing PGD changes validation SSIM from 0.595 to 0.544. In addition, the 3F/5F variants report the same 6.25M parameters despite adding a temporal-attention adapter, and the 5F STD standard deviation (±0.136) is two orders of magnitude smaller than every other entry.

  4. Implausible efficiency numbers. (1) RIS-Fuse is reported at 3624 GFLOPs in 13.8 ms, which exceeds the throughput of an L20; (2) VideoFusion is listed with 0.01M parameters and 0.16 GFLOPs; (3) RT-FuseNet itself takes 1332 ms per frame at 159 GFLOPs with no explanation; and (4) The BLIP-2 captioning, YOLOv8 detection, and text encoding required at inference are excluded from timing, which understates the real cost of a text-guided method.

  5. Prompt ablation design. The “No-prompt” variant (SSIM 0.403) performs far worse than the plain backbone without TAGM or ARB (M1, SSIM 0.535). This contradicts the claim that TAGM reduces to identity under weak modulation, and suggests that text was simply removed from a model trained with text (a train/test mismatch). As a result, the ablation does not isolate the contribution of language guidance.

  6. Evaluation choices. (1) Commonly used fusion metrics (MI, VIF, Q^AB/F, SF, AG) are absent; (2) SSIM is both a training loss and a headline metric; (3) Temporal metrics may reward smoothness: the frame-level A2RNet outperforms most video methods on BiSWE/flowD, and this is not discussed; and (4) All standard deviations are across frames, and many reported margins are small.

  7. Downstream claims. (1) On LLVIP, IR-only input achieves AP50 0.9365 versus 0.8461 for RT-FuseNet; this undercuts the downstream-utility claim and is not discussed, and (2) The segmentation section mixes MSRS and FMB descriptions and uses a different baseline set for FMB (Table 13).

Questions for authors
  1. Please explain why the RT-FuseNet-1F results in Table A1 are identical to the main results despite different training (no PGD) and a different evaluation list. Please also report the true parameter counts for 3F/5F and check the 5F STD entry.
  2. How were the latencies in Table 14 measured? Please report end-to-end latency including caption generation, detection, and text encoding.
  3. How exactly was the No-prompt variant implemented? Was it trained without text, or was text removed at test time?
  4. Can you report a quantitative cross-modal registration error for the registered subset, for example on manually annotated correspondences?
  5. Were text-guided baselines (RIS-Fuse, UP-Fusion, AWM-Fuse) given the same text descriptions as RT-FuseNet?
Reviewer_Qy7DOpenReview ↗
Rating
4(+1)
Confidence
3(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents UAV-VIVID, a large-scale UAV infrared–visible video dataset comprising 375 scenes and 190,978 paired frames under diverse acquisition conditions. It also proposes RT-FuseNet, a text-guided frame-level fusion baseline that combines adaptive residual balancing, semantic modulation, and dual-modal adversarial augmentation. Experiments on multiple benchmarks and downstream perception tasks demonstrate competitive fusion performance and generalization capabilities.

Critical strengths
  1. UAV-VIVID provides a valuable addition to existing infrared–visible video fusion benchmarks. Its 375 scenes cover diverse altitudes, registration conditions, and environmental challenges. The scene-disjoint splits and text annotations further support systematic evaluation of video fusion and cross-condition generalization.

  2. RT-FuseNet combines ARB for complementary cross-modal feature extraction with TAGM for text-guided semantic modulation. The two modules serve distinct yet complementary purposes. Ablation results in Table 16 demonstrate their effectiveness.

Critical weaknesses
  1. The main experiments focus on registered low-altitude sequences, leaving the evaluation protocols for other subsets less clear. The SSIM–flowD trade-off in Table A1 also warrants further discussion.

  2. Table 15 shows that dual-modal PGD augmentation improves fusion performance on clean inputs. The authors should explain the motivation for using clean-source supervision during adversarial training and how the observed improvements should be interpreted.

  3. Why does the registered low-altitude subset serve as the primary fusion benchmark, and how are the remaining subsets intended to support different research tasks?

  4. Table 16 shows that combining ARB and TAGM yields better results than either module alone. The authors should further explain their complementary roles in cross-modal feature interaction and semantic modulation.

  5. Table 17 shows that mismatched prompts reduce fusion quality. The impact of inaccurate automatically generated descriptions on TAGM should be further discussed.

  6. Since RT-FuseNet processes frames independently, how should the reported BiSWE and flowD metrics be interpreted when evaluating its sequence-level behavior?

Questions for authors

See weaknesses

1 comment

Thanks to the authors for their additional explanations and clarifications. They have addressed most of my questions. Considering the value of the proposed UAV-VIVID dataset and its potential to facilitate future video fusion research, I have decided to raise my rating.

Reviewer_JrtKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UAV-VIVID, a dataset comprising continuous UAV infrared and visible light video, which employs a scene-disjoint data partition and provides automatically generated text descriptions. The authors also propose a frame-level fusion baseline, RT-FuseNet, enhanced with differential attention, text-conditional modulation, and PGD training. Experiments evaluated fusion quality, temporal consistency, and downstream perception performance; RT-FuseNet demonstrated superior spatial quality, while VideoFusion performed better on the reported temporal metrics.

Critical strengths
  1. Continuous UAV sequences can be used to study variations in fusion results between adjacent frames, which isolated image pairs cannot fully support. Scene-level partitioning keeps adjacent frames within the same data partition, and the low-altitude subset used for practical evaluation features a fixed development partition and 70 held-out test scenes. Therefore, this dataset remains valuable even when other fusion methods are employed.
  2. The experiments cover multiple datasets, downstream tasks, and both spatial and temporal metrics. Within each evaluation track, comparison methods use the same source training data. RT-FuseNet achieves an SSIM of 0.678 on UAV-VIVID, outperforming the second-best method’s score of 0.654. The paper also reports its relatively weaker temporal performance compared to VideoFusion, clearly highlighting the baseline’s strengths and limitations.
Critical weaknesses
  1. TAGM uses a residual affine update similar to that in Text-IF (Eq. 9), followed by a text-conditioned spatial gate. Existing ablation studies show that using the full TAGM outperforms removing it, but they do not specify the benefits of the added gating mechanism relative to using only residual affine modulation. ARB is also built upon existing differential attention, so its methodological contributions are relatively limited. The model can serve as a useful reference baseline, but the advantages of its specific text modulation design have not yet been fully validated.
  2. A more direct assessment of the quality of the final image pairs in the dataset is needed. The reported intrinsic-calibration reprojection errors and the 25-ms timestamp-matching threshold do not quantify the residual spatial misalignment in the final image pairs, particularly around moving objects. Furthermore, it is difficult to reconcile Table A1 with Table 5: the temporal ablation claims to disable PGD, yet the mean and standard deviation for the seven entries in the 1F row are identical to those of the main model trained with PGD. Until the source of these results is clarified, it is difficult to use this experiment to assess the impact of temporal context itself.
Questions for authors
  1. What benefits does the newly added text conditional gating offer compared to using residual affine modulation alone? Please compare the results using the same backbone, text encoder, prompts, and training settings, including keeping the PGD settings consistent. A comparison with standard text–image cross-attention would also help illustrate the specific advantages of TAGM.
  2. For the low-altitude subset used in the practical evaluation, what specific challenges does UAV-VIVID introduce compared to existing video fusion benchmarks? Can these distinctions be supported by statistical data or analysis, rather than just the list of features in Table 1? Please distinguish between evidence from the practical evaluation subset and the potential uses of the auxiliary higher-altitude sequences and the unregistered lower-altitude subset.
  3. Which sequences in the actual evaluation used pairing by decoded frame order? Please provide the number of affected scenes and image pairs in each partition. For timestamp pairing, please report the distribution of residual timestamp differences after offset correction; a final spatial alignment quality spot check that includes regions of motion would also help assess the benchmark’s reliability.
  4. Was the 1F model in Table A1 indeed trained with PGD disabled and using the exact same list of center frames as the 3F and 5F models? Please specify the checkpoint and evaluation lists and explain why their statistics match those in Table 5 exactly. If corrections are needed, do the conclusions regarding temporal context still hold after the corrections?
1 comment

Dear Reviewer JrtK,

Thank you again for your valuable time and constructive feedback.

We understand that you may have a busy schedule, and we sincerely appreciate your time and effort in reviewing our work. We have carefully addressed your concerns through detailed responses, additional experiments, and further clarifications. We hope that these efforts have adequately resolved the issues raised in your original review.

As the discussion period draws to a close, we would be very grateful if you could kindly revisit our responses. If you find that our clarifications and additional evidence have sufficiently addressed your concerns, we sincerely hope that you would consider updating your rating accordingly.

Thank you once again for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.