- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, a new machine oriented evaluation metric is proposed to solve the problem that the traditional image quality metrics for human vision can not fully reflect the downstream utility of the fused image in the field of visible and infrared image fusion. The author uses FMB, LLVIP and MFAD datasets and 10 fusion methods to form the FuM-18K dataset through manual supplementary detection, segmentation and language understanding annotation, and calculate the task score. The author gives the evaluation algorithm FuM-Net to predict the score, and gives a number of comparative results.
Critical strengths
1. The problem has practical value. The visual attractiveness of the fused image and the utility of the machine task may be inconsistent, so it is of clear application significance to establish the evaluation for this difference.
2. The source image revenue benchmark is explanatory. Compared with only reporting the absolute task performance of the fused image, can directly indicate whether the fusion is better than the two single-mode inputs, allowing negative returns.
3. Mark resources with potential value. Although there are only 1892 independent source scenarios in the dataset, the workload of unified supplementary detection, segmentation, VQA and caption annotation is worthy of recognition.
4. Language understanding annotation is a useful extension of common fusion datasets.
Critical weaknesses
1. Pages 7 and 8 only state an 8:1:1 split. They do not say whether the split is performed by source pair, scene/video sequence, or fused image. Ten fused outputs from one source pair share scene content, source images, and annotations. If the 18,920 fused images were split independently, near-duplicate scenes could appear in both training and test sets and inflate correlations. The authors should provide a source-pair/sequence-level split manifest and report whether the fusion models or downstream evaluators were trained on the evaluated scenes.
2. Lines 426–427 claim substantial superiority across all tasks. Table 1 does not support that claim: for captioning, FuM-Net obtains 0.632/0.646, while Q-Align obtains 0.661/0.663 and LIQE obtains 0.656/0.650. Segmentation SRCC is 0.933 versus 0.932 for RA-MIQA, the difference between which is only 0.001.
3. SD is cited to Rao (1997), whose record is In-fibre Bragg grating sensors, a fiber-sensor paper unrelated to the standard-deviation fusion metric.
4. The detection response averages matched GT-object IoUs and assigns zero to unmatched GT objects, but it does not penalize unmatched predicted boxes (Appendix A.2, lines 875–887). A detector producing many false-positive boxes could therefore obtain a high score. The authors should compare this response with AP, Precision, and Recall and specify confidence thresholds, NMS, and matching details.
5. The improvement of ablation part is very limited. The F-only ablation already obtains 0.839/0.834, whereas the full model obtains 0.848/0.841, an increase of only 0.009/0.007. Without multiple seeds or source-clustered confidence intervals, the stability of this gain is unclear.
Questions for authors
1. What is the unit of the 8:1:1 split? Please provide information of source-pair, original-dataset, and sequence-level manifests, demonstrate that related fused images and neighboring frames do not cross splits, and list training-data overlap for fusion models and evaluators.
2. Why are false-positive detections not penalized? Do AP, Precision/Recall, or false-positive-sensitive alternatives preserve the score and FuM-Net ranking?
3. For the 10,392 retained VQA questions and 1,892 captions, what were the annotation effort, correction rate, and final error rate?
4. Please explain the Rao (1997) citation links.
10 comments
Response to Reviewer kiSz (Part 2)
Weakness 2: Performance of FuM-Net
Response: Our comparison considers both prediction performance and computational efficiency. FuM-Net achieves the highest overall PLCC and SRCC in Table 1 (in manuscript, ). Its segmentation performance is comparable to RA-MIQA. Table 2 (in manuscript, ) shows that FuM-Net uses approximately 45% fewer trainable parameters and runs approximately 2.85 times as fast as RA-MIQA. Although LIQE and Q-Align achieve higher captioning correlations, FuM-Net provides higher overall correlations with lower computational cost. These results demonstrate a favorable trade-off between prediction performance and efficiency.
Weakness 3 & Question 4: Clarification of the Rao (1997) citation
Response: We apologize for this citation error. We have corrected the reference [Ref4] in the revised manuscript and . The error resulted from insufficient verification of the correspondence between the cited reference and the SD metric.
[Ref4] Shi, Wenzhong, et al. Wavelet-based image fusion and quality assessment. JAG, 2005.
Response to Reviewer kiSz (Part 3.a)
Weakness 4 & Question 2: Construction and validation of FuM-Score
Response: Object detection score. In detection, our goal is to measure how many objects are correctly detected and how accurately they are localized. Accordingly, we focus on assessing fusion gains in object coverage and localization quality relative to the source modality. To further validate this design, we examine the correlations between FuM-Score and conventional detection metrics.
On the FuM-18K test set, we compare fusion-method rankings based on with conventional detection metrics. As shown in Table 3, the SRCC exceeds 0.85 for both mAP50 and mAP50:95, and 0.7 for Recall. Since AP accounts for false positives, these results show strong agreement with fusion-method rankings obtained using a false-positive-sensitive detection metric. We also compare the rankings produced by FuM-Net predictions with the same detection metrics. As shown in Table 3, the SRCC values with mAP50, mAP50:95 and Recall all exceed 0.8. This indicates that FuM-Net predictions also reflect fusion-method rankings under standard detection evaluation. This can be attributed to the fact that some fusion methods detect more true targets while also producing more background and low-overlap false positives, increasing the FP-to-TP ratio. Consequently, these methods tend to achieve higher Recall and mAP despite having lower Precision. In contrast, our detection score primarily reflects target detection and localization quality, rather than penalizing false positives to the same extent as Precision. Therefore, on FuM-18K and its downstream detection models, our score exhibits stronger rank correlations with AP and Recall but a weaker correlation with Precision.
Table 3. SRCC / PLCC between fusion-method scores based on or FuM-Net predictions and conventional detection metrics on the FuM-18K test set.
| mAP50 | mAP50:95 | Recall | Precision | |
|---|---|---|---|---|
| 0.8909 / 0.7649 | 0.9152 / 0.8248 | 0.7697 / 0.7703 | -0.2727 / -0.3159 | |
| 0.8545 / 0.8986 | 0.8182 / 0.9161 | 0.8788 / 0.8109 | -0.4303 / -0.4659 |
Response to Reviewer kiSz (Part 1)
Weakness 1 & Question 1: The data split of FuM-18K
Response: Thanks for your suggestion. Data split. FuM-18K is split by original infrared-visible source pair. Each source pair and all ten fused images generated from it are assigned to the same split. Specifically, we first divide the FMB [Ref1], LLVIP [Ref2], and MFAD [Ref3] subsets separately into training, validation, and test sets at an approximate ratio of 8:1:1, and then merge the corresponding splits. This preserves the representation of each source dataset across the three splits. Table 1 reports the resulting distribution. These details have been added to Appendix D.2 of the revised manuscript and highlighted in blue.
Table 1. Distribution of source pairs across the training, validation, and test splits of FuM-18K.
Training data overlap. All fusion models used to construct FuM-18K were trained without access to the evaluated scenes. For fusion methods originally trained on FMB, LLVIP, or MFAD, we retrained them on a separate dataset excluding these scenes. The downstream evaluators use existing pretrained checkpoints. We verified that their training data did not include FMB, LLVIP, or MFAD. All checkpoints were kept fixed throughout response collection. We used three evaluators per task and averaged their responses to reduce dependence on a single model. Table 2 summarizes their training datasets or checkpoints and we have added the evaluator settings and the table to Appendix B.1 (highlighted in blue).
Table 2. Task evaluators and their training datasets or pretrained checkpoints.
[Ref1] Liu, Jinyuan, et al. Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation. ICCV, 2023.
[Ref2] Jia, Xinyu, et al. LLVIP: A visible-infrared paired dataset for low-light vision. ICCVW, 2021.
[Ref3] Hu, Ke, et al. Ei$^2$det: Edge-guided illumination-aware interactive learning for visible-infrared object detection. TCSVT, 2025.