acceptodds
Under review as a conference paper at ICLR 2027

Assessing Infrared-Visible Fusion Towards Machine Perception and Understanding Preferences: Benchmark, Dataset, and Baseline

Abstract

Despite substantial progress in infrared-visible image fusion (IVIF) over the past decades, fused-image evaluation still relies heavily on metrics adapted from generic image quality assessment, which primarily cater to human perceptual preference. Machine-centric evaluation, by contrast, remains underexplored due to the lack of dedicated benchmarks. To address this limitation, we formalize machine preference as the utility of fused images for representative downstream perception and understanding tasks. We then introduce FuM-Score, which jointly quantifies such preference by aggregating fusion gains relative to the infrared and visible sources across these tasks under standardized evaluation protocols. Based on these, we subsequently establish FuM-18K, to our knowledge, the first large-scale dataset for machine-centric assessment in IVIF, comprising over 18,000 images spanning diverse scenes and representative fusion models, along with fine-grained downstream task annotations. Finally, an effective FuM-Net is built to leverage fusion features together with auxiliary source references, achieving automatic FuM-Score prediction for quality assessment. Extensive experiments validate the efficacy of our benchmark in capturing machine preference and demonstrate the superiority of FuM-Net over recent state-of-the-art methods.

Primary area:
datasets and benchmarks
Keywords:
infrared-visible image fusion, image quality assessment, machine-oriented image quality assessment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kiSzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, a new machine oriented evaluation metric is proposed to solve the problem that the traditional image quality metrics for human vision can not fully reflect the downstream utility of the fused image in the field of visible and infrared image fusion. The author uses FMB, LLVIP and MFAD datasets and 10 fusion methods to form the FuM-18K dataset through manual supplementary detection, segmentation and language understanding annotation, and calculate the task score. The author gives the evaluation algorithm FuM-Net to predict the score, and gives a number of comparative results.

Critical strengths

1. The problem has practical value. The visual attractiveness of the fused image and the utility of the machine task may be inconsistent, so it is of clear application significance to establish the evaluation for this difference.

2. The source image revenue benchmark is explanatory. Compared with only reporting the absolute task performance of the fused image, GtG_t can directly indicate whether the fusion is better than the two single-mode inputs, allowing negative returns.

3. Mark resources with potential value. Although there are only 1892 independent source scenarios in the dataset, the workload of unified supplementary detection, segmentation, VQA and caption annotation is worthy of recognition.

4. Language understanding annotation is a useful extension of common fusion datasets.

Critical weaknesses

1. Pages 7 and 8 only state an 8:1:1 split. They do not say whether the split is performed by source pair, scene/video sequence, or fused image. Ten fused outputs from one source pair share scene content, source images, and annotations. If the 18,920 fused images were split independently, near-duplicate scenes could appear in both training and test sets and inflate correlations. The authors should provide a source-pair/sequence-level split manifest and report whether the fusion models or downstream evaluators were trained on the evaluated scenes.

2. Lines 426–427 claim substantial superiority across all tasks. Table 1 does not support that claim: for captioning, FuM-Net obtains 0.632/0.646, while Q-Align obtains 0.661/0.663 and LIQE obtains 0.656/0.650. Segmentation SRCC is 0.933 versus 0.932 for RA-MIQA, the difference between which is only 0.001.

3. SD is cited to Rao (1997), whose record is In-fibre Bragg grating sensors, a fiber-sensor paper unrelated to the standard-deviation fusion metric.

4. The detection response averages matched GT-object IoUs and assigns zero to unmatched GT objects, but it does not penalize unmatched predicted boxes (Appendix A.2, lines 875–887). A detector producing many false-positive boxes could therefore obtain a high score. The authors should compare this response with AP, Precision, and Recall and specify confidence thresholds, NMS, and matching details.

5. The improvement of ablation part is very limited. The F-only ablation already obtains 0.839/0.834, whereas the full model obtains 0.848/0.841, an increase of only 0.009/0.007. Without multiple seeds or source-clustered confidence intervals, the stability of this gain is unclear.

Questions for authors

1. What is the unit of the 8:1:1 split? Please provide information of source-pair, original-dataset, and sequence-level manifests, demonstrate that related fused images and neighboring frames do not cross splits, and list training-data overlap for fusion models and evaluators.

2. Why are false-positive detections not penalized? Do AP, Precision/Recall, or false-positive-sensitive alternatives preserve the score and FuM-Net ranking?

3. For the 10,392 retained VQA questions and 1,892 captions, what were the annotation effort, correction rate, and final error rate?

4. Please explain the Rao (1997) citation links.

10 comments
AuthorsOpenReview ↗

Response to Reviewer kiSz (Part 1)

Weakness 1 & Question 1: The data split of FuM-18K

Response: Thanks for your suggestion. Data split. FuM-18K is split by original infrared-visible source pair. Each source pair and all ten fused images generated from it are assigned to the same split. Specifically, we first divide the FMB [Ref1], LLVIP [Ref2], and MFAD [Ref3] subsets separately into training, validation, and test sets at an approximate ratio of 8:1:1, and then merge the corresponding splits. This preserves the representation of each source dataset across the three splits. Table 1 reports the resulting distribution. These details have been added to Appendix D.2 of the revised manuscript and highlighted in blue\textcolor{#0F9ED5}{\text{highlighted in blue}}.

Table 1. Distribution of source pairs across the training, validation, and test splits of FuM-18K.

Sub-datasetTotal (Pairs)Train (Pairs)Validation (Pairs)Test (Pairs)
FMB [Ref1]14921193149150
LLVIP [Ref2]2502002525
MFAD [Ref3]1501201515
FuM-18K18921513189190

Training data overlap. All fusion models used to construct FuM-18K were trained without access to the evaluated scenes. For fusion methods originally trained on FMB, LLVIP, or MFAD, we retrained them on a separate dataset excluding these scenes. The downstream evaluators use existing pretrained checkpoints. We verified that their training data did not include FMB, LLVIP, or MFAD. All checkpoints were kept fixed throughout response collection. We used three evaluators per task and averaged their responses to reduce dependence on a single model. Table 2 summarizes their training datasets or checkpoints and we have added the evaluator settings and the table to Appendix B.1 (highlighted in blue\textcolor{#0F9ED5}{\text{highlighted in blue}}).

Table 2. Task evaluators and their training datasets or pretrained checkpoints.

TaskModelTraining dataset/Checkpoint
Object detectionYOLOv11COCO
Object detectionRT-DETRCOCO
Object detectionFaster R-CNNCOCO
Semantic segmentationSegFormerADE20K
Semantic segmentationX-DecoderCOCO 2017
Semantic segmentationDeepLabV3+Cityscapes
VQA & CaptionQwen3-VL-8BQwen3-VL-8B-Instruct
VQA & CaptionGLM-4.1V-9BGLM-4.1V-9B-Base
VQA & CaptionInternVL3-8BInternVL3-8B

[Ref1] Liu, Jinyuan, et al. Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation. ICCV, 2023.

[Ref2] Jia, Xinyu, et al. LLVIP: A visible-infrared paired dataset for low-light vision. ICCVW, 2021.

[Ref3] Hu, Ke, et al. Ei$^2$det: Edge-guided illumination-aware interactive learning for visible-infrared object detection. TCSVT, 2025.

AuthorsOpenReview ↗

Response to Reviewer kiSz (Part 2)

Weakness 2: Performance of FuM-Net

Response: Our comparison considers both prediction performance and computational efficiency. FuM-Net achieves the highest overall PLCC and SRCC in Table 1 (in manuscript, highlighted in orange\textcolor{#E97132}{\text{highlighted in orange}} ). Its segmentation performance is comparable to RA-MIQA. Table 2 (in manuscript, highlighted in orange\textcolor{#E97132}{\text{highlighted in orange}}) shows that FuM-Net uses approximately 45% fewer trainable parameters and runs approximately 2.85 times as fast as RA-MIQA. Although LIQE and Q-Align achieve higher captioning correlations, FuM-Net provides higher overall correlations with lower computational cost. These results demonstrate a favorable trade-off between prediction performance and efficiency.

Weakness 3 & Question 4: Clarification of the Rao (1997) citation

Response: We apologize for this citation error. We have corrected the reference [Ref4] in the revised manuscript and highlighted the change in blue\textcolor{#0F9ED5}{\text{highlighted the change in blue}}. The error resulted from insufficient verification of the correspondence between the cited reference and the SD metric.

[Ref4] Shi, Wenzhong, et al. Wavelet-based image fusion and quality assessment. JAG, 2005.

AuthorsOpenReview ↗

Response to Reviewer kiSz (Part 3.a)

Weakness 4 & Question 2: Construction and validation of FuM-Score

Response: Object detection score. In detection, our goal is to measure how many objects are correctly detected and how accurately they are localized. Accordingly, we focus on assessing fusion gains in object coverage and localization quality relative to the source modality. To further validate this design, we examine the correlations between FuM-Score and conventional detection metrics.

On the FuM-18K test set, we compare fusion-method rankings based on QDet\mathcal{Q}_{Det} with conventional detection metrics. As shown in Table 3, the SRCC exceeds 0.85 for both mAP50 and mAP50:95, and 0.7 for Recall. Since AP accounts for false positives, these results show strong agreement with fusion-method rankings obtained using a false-positive-sensitive detection metric. We also compare the rankings produced by FuM-Net predictions Q^Det\widehat{\mathcal{Q}}_{Det} with the same detection metrics. As shown in Table 3, the SRCC values with mAP50, mAP50:95 and Recall all exceed 0.8. This indicates that FuM-Net predictions also reflect fusion-method rankings under standard detection evaluation. This can be attributed to the fact that some fusion methods detect more true targets while also producing more background and low-overlap false positives, increasing the FP-to-TP ratio. Consequently, these methods tend to achieve higher Recall and mAP despite having lower Precision. In contrast, our detection score primarily reflects target detection and localization quality, rather than penalizing false positives to the same extent as Precision. Therefore, on FuM-18K and its downstream detection models, our score exhibits stronger rank correlations with AP and Recall but a weaker correlation with Precision.

Table 3. SRCC / PLCC between fusion-method scores based on QDet\mathcal{Q}_{\mathrm{Det}} or FuM-Net predictions Q^Det\widehat{\mathcal{Q}}_{Det} and conventional detection metrics on the FuM-18K test set.

mAP50mAP50:95RecallPrecision
QDet\mathcal{Q}_{Det}0.8909 / 0.76490.9152 / 0.82480.7697 / 0.7703-0.2727 / -0.3159
Q^Det\widehat{\mathcal{Q}}_{Det}0.8545 / 0.89860.8182 / 0.91610.8788 / 0.8109-0.4303 / -0.4659
Reviewer_isQfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work aims to address the inconsistency between existing infrared-visible image fusion quality metrics and downstream machine vision performance by introducing FuM-Score, a new evaluation benchmark oriented toward machine perception and understanding, together with the FuM-18K dataset covering object detection, semantic segmentation, visual question answering, and image captioning. Based on this benchmark, the authors further propose FuM-Net, which predicts quality scores through feature interaction between the fused image and the source images. Experiments comparing existing evaluation methods, cross-evaluator analysis, and ablation studies are conducted to validate the effectiveness of the proposed framework.

Critical strengths
  1. This work is motivated by the gap between fusion quality evaluation and downstream task requirements, addressing a practically relevant problem. Compared with conventional metrics based mainly on image statistics or perceptual quality, it evaluates fused images from the perspective of machine task performance, which is an interesting direction.

  2. The paper provides relatively comprehensive work on benchmark evaluation and dataset construction. FuM-18K covers multiple fusion methods and four downstream tasks, and the authors further analyze evaluation differences across tasks and the influence of different evaluators on the resulting scores.

Critical weaknesses
  1. The construction of FuM-Score could be further discussed. For example, the object detection score does not directly account for false positives, while VQA and image captioning depend on specific task formulations and evaluation metrics. These choices may affect the relative ranking of fusion methods, and the current validation of the score design itself remains limited.

  2. The evaluation of generalization is still insufficient. FuM-18K is generated from a limited number of source image pairs, while it is unclear whether the data split is scene-independent. In addition, although the paper analyzes fusion methods not used for dataset construction, it does not report FuM-Net predictions for these unseen methods, leaving its generalization ability insufficiently validated.

Questions for authors
  1. The authors are encouraged to further clarify the data split of FuM-18K, particularly whether it is performed based on the original image pairs to ensure scene independence between the training and test sets.

  2. For the object detection score, the authors could further explain why false positives are not directly considered and discuss whether the ranking of different fusion methods remains stable when conventional detection metrics are used.

  3. In Table 4, the performance gain from introducing the source images is relatively limited. A comparison with simpler feature-fusion structures under the same pretrained features and input settings would help further justify the necessity of the proposed module.

9 comments
AuthorsOpenReview ↗

Response to Reviewer isQf (Part 1.a)

Weakness 1 & Question 2: Construction and validation of FuM-Score

Response: Object detection score. In detection, our goal is to measure how many objects are correctly detected and how accurately they are localized. Accordingly, we focus on assessing fusion gains in object coverage and localization quality relative to the source modality. To further validate this design, we examine the correlations between FuM-Score and conventional detection metrics.

On the FuM-18K test set, we compare fusion-method rankings based on QDet\mathcal{Q}_{Det} with conventional detection metrics. As shown in Table 1, the SRCC exceeds 0.85 for both mAP50 and mAP50:95, and 0.7 for Recall. Since AP accounts for false positives, these results show strong agreement with fusion-method rankings obtained using a false-positive-sensitive detection metric. We also compare the rankings produced by FuM-Net predictions $ \widehat{\mathcal{Q}}_{Det} $ with the same detection metrics. As shown in Table 1, the SRCC values with mAP50, mAP50:95 and Recall all exceed 0.8. This indicates that FuM-Net predictions also reflect fusion-method rankings under standard detection evaluation. This can be attributed to the fact that some fusion methods detect more true targets while also producing more background and low-overlap false positives, increasing the FP-to-TP ratio. Consequently, these methods tend to achieve higher Recall and mAP despite having lower Precision. In contrast, our detection score primarily reflects target detection and localization quality, rather than penalizing false positives to the same extent as Precision. Therefore, on FuM-18K and its downstream detection models, our score exhibits stronger rank correlations with AP and Recall but a weaker correlation with Precision.

This is because on the current FuM-18K, some fusion methods have detected more true targets, while also generating more background and low overlap false positives , resulting in an increase in the ratio of FP to TP. Therefore, these methods have higher recall and mAP, but lower precision. Our detection score mainly measures whether the real target is detected and the quality of localization. Therefore, in this dataset and downstream task models, our score has a stronger correlation with AP and Recall, but differs from Precision.

Table 1. SRCC / PLCC between fusion-method scores based on QDet\mathcal{Q}_{\mathrm{Det}} or FuM-Net predictions $ \widehat{\mathcal{Q}}_{Det} $ and conventional detection metrics on the FuM-18K test set.

mAP50mAP50:95RecallPrecision
$ \mathcal{Q}_{Det} $0.8909 / 0.76490.9152 / 0.82480.7697 / 0.7703-0.2727 / -0.3159
$ \widehat{\mathcal{Q}}_{Det} $0.8545 / 0.89860.8182 / 0.91610.8788 / 0.8109-0.4303 / -0.4659
AuthorsOpenReview ↗

Response to Reviewer isQf (Part 1.b)

Weakness 1 & Question 2: Construction and validation of FuM-Score

Following your valuable suggestion, we further evaluate a variant with an explicit false-positive penalty. We keep the original matching rules and add the number of unmatched predictions to the denominator:

\frac{\sum_{i=1}^{N_{\mathrm{GT}}}s_i} {N_{\mathrm{TP}}+0.5N_{\mathrm{FN}}+0.5N_{\mathrm{FP}}}, \qquad s_i= \begin{cases} \operatorname{IoU}(b_i,\hat b_i), & \text{if the match is valid},\\\\ 0, & \text{otherwise}. \end{cases}$$ Here, $N_{\mathrm{GT}} = N_{\mathrm{TP}} + N_{\mathrm{FN}}$ denotes the number of ground-truth objects. $N_{\mathrm{TP}}$, $N_{\mathrm{FN}}$, and $N_{\mathrm{FP}}$ denote the numbers of valid matches, missed ground-truth objects, and unmatched predictions, respectively. This variant preserves the original response when no false positives occur. We apply it to infrared, visible, and fused images, then use the same evaluator aggregation and source-reference calculation to obtain $\mathcal{Q}_{\mathrm{Det}}^{\mathrm{FP}}$. **Table 2. SRCC / PLCC between fusion-method scores and conventional detection metrics before and after introducing the false-positive penalty on the FuM-18K test set.** | Score | mAP50 | mAP50:95 | Recall | Precision | | ------------------------------------------ | --------------- | --------------- | --------------- | ----------------- | | $\mathcal{Q}_{\mathrm{Det}}$ | 0.8909 / 0.7649 | 0.9152 / 0.8248 | 0.7697 / 0.7703 | -0.2727 / -0.3159 | | $\mathcal{Q}_{\mathrm{Det}}^{\mathrm{FP}}$ | 0.636 / 0.436 | 0.733 / 0.616 | 0.588 / 0.364 | 0.152 / 0.154 | **As shown in Table 2**, the variant improves both SRCC and PLCC with Precision from negative to positive. On FuM-18K, some fusion methods detect more true objects and achieve higher Recall and mAP. They also produce more background and low-overlap false positives, resulting in a higher FP/TP ratio and lower Precision. The FP variant gives greater weight to these errors and reduces the relative emphasis on object coverage. Its rankings therefore move closer to Precision, while agreement with AP and Recall decreases. Besides, we can see that the incorporation of FP penalty improves agreement with Precision. However, considering the goal of this work is to measure how many objects are correctly detected and how accurately they are localized, alongside the marginal improvements in Precision with false-positive penalty but significant drop in mAP and Recall, we exclude this in our design. **VQA and image captioning scores.** We further examine the consistency of $\mathcal{Q}_{\mathrm{VQA}}$ and $\mathcal{Q}_{\mathrm{Cap}}$ with other commonly used evaluation metrics on FuM-18K. **As shown in Table 3**, our scores show good agreement with VQA Recall, captioning ROUGE-L, and BLEU-4. These results support the consistency of our scores with alternative evaluation metrics. **Table 3. SRCC / PLCC between FuM-Score and alternative evaluation metrics on FuM-18K.** | **Score** | **Evaluation metric** | **SRCC / PLCC** | | ---------------------------- | --------------------- | --------------- | | $\mathcal{Q}_{\mathrm{VQA}}$ | Recall | 0.6687 / 0.9328 | | $\mathcal{Q}_{\mathrm{Cap}}$ | ROUGE-L | 0.7590 / 0.7502 | | $\mathcal{Q}_{\mathrm{Cap}}$ | BLEU-4 | 0.6221 / 0.6440 |
AuthorsOpenReview ↗

Response to Reviewer isQf (Part 2)

Weakness 2 & Question 1: Data split and generalization of FuM-Net

Response: Thanks for your suggestion. Data split. FuM-18K is split by original infrared-visible source pair. Each source pair and all ten fused images generated from it are assigned to the same split. Specifically, we first divide the FMB [Ref1], LLVIP [Ref2], and MFAD [Ref3] subsets separately into training, validation, and test sets at an approximate ratio of 8:1:1, and then merge the corresponding splits. This preserves the representation of each source dataset across the three splits. Table 4 reports the resulting distribution. Besides, we also add these details in our Appendix D.2 (highlighted in blue\textcolor{#0F9ED5}{\text{highlighted in blue}}).

Table 4. Distribution of source pairs across the training, validation, and test splits of FuM-18K.

Sub-datasetTotal (Pairs)Train (Pairs)Validation (Pairs)Test (Pairs)
FMB [Ref1]14921193149150
LLVIP [Ref2]2502002525
MFAD [Ref3]1501201515
FuM-18K18921513189190

Generalization to unseen fusion methods. We evaluate the frozen FuM-Net on fusion methods not used during its training, using source pairs from the original test set. As shown in Table 5, its overall predictions show consistent agreement with FuM-Score across the unseen methods. These results support the generalization of FuM-Net to unseen fusion methods without further training. We also add these details in our Appendix D.2 (highlighted in blue\textcolor{#0F9ED5}{\text{highlighted in blue}}).

Table 5. Prediction performance of FuM-Net on unseen fusion methods. Results are reported as SRCC / PLCC.

Fusion methodOverallDetSegVQACap
Text-IF [Ref4]0.8307 / 0.83910.8099 / 0.81470.9263 / 0.93320.6190 / 0.59750.6013 / 0.5959
FreeFusion [Ref5]0.8276 / 0.83160.8261 / 0.81670.8996 / 0.91640.6029 / 0.60500.6717 / 0.6594
UniFusion [Ref6]0.8190 / 0.82860.8050 / 0.79230.9248 / 0.93930.6173 / 0.61050.6253 / 0.6073
DCEvo [Ref7]0.8034 / 0.81660.8022 / 0.79820.9312 / 0.93930.6163 / 0.60950.6766 / 0.6598
DPOFusion [Ref8]0.7975 / 0.80500.8344 / 0.83260.8057 / 0.86160.5303 / 0.52240.6426 / 0.6213

[Ref1] Liu, Jinyuan, et al. Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation. ICCV, 2023.

[Ref2] Jia, Xinyu, et al. LLVIP: A visible-infrared paired dataset for low-light vision. ICCVW, 2021.

[Ref3] Hu, Ke, et al. Ei$^2$det: Edge-guided illumination-aware interactive learning for visible-infrared object detection. TCSVT, 2025.

[Ref4] Yi, Xunpeng, et al, Text-if: Leveraging semantic text guidance for degradation-aware and interactive image fusion. CVPR, 2024.

[Ref5] Zhao, Wenda, et al, FreeFusion: Infrared and visible image fusion via cross reconstruction learning. TPAMI, 2025.

[Ref6] Li, Xingyuan, et al. UniFusion: A unified image fusion framework with robust representation and source-aware preservation. CVPR, 2026.

[Ref7] Liu, Jinyuan, et al. Dcevo: Discriminative cross-dimensional evolutionary learning for infrared and visible image fusion. CVPR, 2025.

[Ref8] Su, Weijian, et al. Fusion in your way: Aligning image fusion with heterogeneous demands via direct preference optimization. CVPR, 2026.

Reviewer_KxgzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies machine-centric quality assessment for infrared–visible image fusion. It introduces FuM-Score based on downstream task performance and gains over source images, and constructs FuM-18K from 1,892 source pairs and 18,920 fusion results. Four tasks are considered: object detection, semantic segmentation, VQA, and image captioning. FuM-Net is trained to predict the task-specific scores, with evaluation against existing quality assessment methods.

Critical strengths

The proposed dataset with four tasks is interesting.

Critical weaknesses
  1. The motivation relies on an inaccurate characterization of existing IVIF evaluation. The abstract states that evaluation still relies heavily on traditional image quality metrics, while the introduction emphasizes visual aesthetics. However, many papers in recent years already evaluate fusion through downstream tasks, particularly detection and segmentation. Apart from including four tasks, the paper does not sufficiently explain what fundamentally distinguishes this benchmark from existing task-based evaluation. The claim of the first large-scale machine-centric dataset also needs clearer qualification. FuM-18K contains 18,920 fusion results, but only 1,892 source pairs, which should be clearly distinguished when discussing dataset scale.

  2. The experimental setting is not sufficiently clear. The paper lists the downstream evaluators but does not explain whether they were trained or fine-tuned using fused images. This is important for interpreting the results and ensuring fair comparisons. Moreover, apart from including four tasks, it is unclear what this benchmark adds to existing detection- and segmentation-based evaluation. These issues make the practical significance of the proposed evaluation unclear.

  3. The paper does not sufficiently explain why generating a fused image is advantageous for machine perception compared with feature-level fusion.

Questions for authors
  1. Many recent IVIF papers already use downstream tasks for evaluation. Beyond extending evaluation to four tasks, what is the main difference from existing detection- and segmentation-based assessment? How is the claimed first machine-centric dataset distinguished from existing task-oriented datasets?

  2. Were the detection and segmentation evaluators trained or fine-tuned on fused images? Were the training data and procedures consistent across source modalities and fusion methods?

  3. Under what conditions does the gap illustrated in Figure 1(a) hold? How sensitive are the conclusions to the downstream models and their training settings? I don't think this gap is always true.

  4. For machine perception, what is the practical advantage of generating a fused image compared with feature-level fusion? Please clarify the scope and practical relevance of the proposed assessment.

9 comments
AuthorsOpenReview ↗

Response to Reviewer Kxgz (Part 1)

Weakness 1 & Question 1: The difference between existing task-adaptive dataset and assessment

Response: We agree that recent task-adaptive IVIF methods already evaluate fused images through downstream tasks. However, these methods typically optimize fusion for a specific task, and strong performance on that task does not necessarily translate into comparable benefits for other tasks. More importantly, the performance on perception (detection or segmentation) tasks also cannot represent the utility for machine understanding (e.g. VQA and caption). A comprehensive assessment requires multiple task models and their associated training and testing procedures, making the evaluation cumbersome and time-consuming. In contrast, our contribution is to formalize machine preference in IVIF and construct a unified benchmark that quantifies it through task utility and source-referenced fusion gains across representative perception and understanding tasks. FuM-18K and FuM-Score provide a unified quality measure for this purpose. Besides, we also introduce a more lightweight baseline, FuM-Net, which learns to predict these scores directly without running multiple downstream evaluators at inference.

Dataset scale. To our knowledge, FuM-18K is the first large-scale dataset for machine preference quality assessment in IVIF (following Table 1), providing aligned task annotations and source-referenced quality labels. FuM-18K contains 1,892 source pairs and 18,920 fused images generated by ten fusion methods. As shown, FuM-18K includes substantially more source pairs and fused images than the existing IVIF quality assessment datasets listed, while focusing on machine preference.

Table 1. Scale and assessment focus of representative IVIF quality assessment datasets.

DatasetSource pairsFused imagesAssessment focus
VIFB [Ref1]21420Traditional fusion metrics
VISTA [Ref2]2004,000Human preference
EVAFusion [Ref3]8509,350Human preference
FuM-18K1,89218,920Machine preference

[Ref1] Zhang, Xingchen, et al, VIFB: A visible and infrared image fusion benchmark. CVPRW, 2020.

[Ref2] Chang, Zhihao, et al, Semantic-relation transformer for visible and infrared fused image quality assessment. Inform Fusion, 2023.

[Ref3] Liu, Jinyuan, et al. Bridging human evaluation to infrared and visible image fusion. CVPR, 2026.

AuthorsOpenReview ↗

Response to Reviewer Kxgz (Part 2)

Weakness 1 & Question 3: Ranking conflicts in Figure 1(a) and evaluator sensitivity

Response: Figure 1(a) illustrates a ranking conflict between existing quality metrics and downstream task utility. To quantify its frequency, for each source pair in FuM-18K, we compare the ten fused images in all (102)=45\binom{10}{2}=45 unordered pairs. Let mj(I)m_j(I) denote the score of quality metric jj for image II. For two fused images IfaI_f^a and IfbI_f^b from the same source pair, a ranking conflict occurs when the metric and evaluator ee on task tt prefer different images:

\cdot \left[r_t^{(e)}(I_f^a)-r_t^{(e)}(I_f^b)\right],$$ where $r_t^{(e)}$ is the response of evaluator $e$ on task $t$. A negative value indicates a conflict, and a positive value indicates agreement. Let $n_i^{\mathrm{con}}$ and $n_i^{\mathrm{agr}}$ denote the numbers of conflicts and agreements for source pair $i$. Its conflict rate is: $$d_i = \frac{n_i^{\mathrm{con}}} {n_i^{\mathrm{con}}+n_i^{\mathrm{agr}}}, \qquad \bar d = \frac{1}{|V|}\sum_{i\in V} d_i.$$ We average these rates across source pairs for each quality metric and evaluator. As shown in $\textcolor{#0F9ED5}{\text{Figure 6 (revised manuscript)}}$, the Aggregate row uses the response averaged over the three evaluators before computing the conflict rate. Most rates are around **40%–50%** across the four tasks, showing frequent disagreement between existing quality metrics and downstream task utility. The three evaluators produce **similar rates in most cases**. This suggests limited sensitivity to the choice among the tested downstream models under their fixed evaluation settings. We also add the details and figure in our **Appendix B.3** ($\textcolor{#0F9ED5}{\text{highlighted in blue}}$).
AuthorsOpenReview ↗

Response to Reviewer Kxgz (Part 3)

Weakness 2 & Question 2: The experimental setting

Response: First, we did not train or fine-tune the detection and segmentation evaluators on the fused images in FuM-18K. Second, the evaluator configuration was kept consistent across the source modalities and fusion methods. Each evaluator used the same pretrained checkpoint, preprocessing, and scoring rules for infrared, visible, and fused images.

We adopted existing checkpoints and held them fixed throughout response collection. We used three evaluators per task and averaged their responses to reduce dependence on a single model. Table 2 provides details for the evaluator setup and we have added the evaluator settings and the table to Appendix B.1 (highlighted in blue\textcolor{#0F9ED5}{\text{highlighted in blue}}).

Table 2. Task evaluators and their training datasets or pretrained checkpoints.

TaskModelTraining dataset/Checkpoint
Object detectionYOLOv11COCO
Object detectionRT-DETRCOCO
Object detectionFaster R-CNNCOCO
Semantic segmentationSegFormerADE20K
Semantic segmentationX-DecoderCOCO 2017
Semantic segmentationDeepLabV3+Cityscapes
VQA & CaptionQwen3-VL-8BQwen3-VL-8B-Instruct
VQA & CaptionGLM-4.1V-9BGLM-4.1V-9B-Base
VQA & CaptionInternVL3-8BInternVL3-8B

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.