acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Infrared-Visible Image Fusion via Semantic Diagnosis and Reward-Driven Optimization

Abstract

Infrared and visible (IR-VIS) image fusion aims to integrate thermal target saliency from IR images and textural details from VIS images into a single fused image. Existing methods mainly rely on low-level visual fidelity objectives, emphasizing image sharpness and structural consistency while overlooking semantic preservation and task-relevant cues. Moreover, the lack of ground-truth fusion labels and scene-dependent trade-offs among thermal saliency, texture details, structural consistency, and artifact suppression make fixed loss functions inadequate for adaptive optimization. To address these limitations, we propose a semantic-diagnosis- and reward-driven framework that combines high-level semantic guidance with candidate-based reward optimization. Specifically, we introduce a frozen DINOv3-ConvNeXt to provide hierarchical visual priors and global visual representations, together with a lightweight adaptation branch for local texture and detail enhancement. Building on these representations, we leverage a frozen Qwen3-VL-Embedding to perform coarse-to-fine semantic diagnosis on intermediate fusion states and progressively guide their refinement. To further enable adaptive optimization without fusion labels, we generate multiple fusion candidates through perturbation sampling and evaluate their relative quality using multidimensional rewards. We then employ an advantage-weighted objective to optimize the candidate generation policy, thereby learning fusion policies through candidate comparison and reward optimization. Experimental results demonstrate that our method consistently outperforms existing approaches in both quantitative metrics and visual quality across multiple public benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
infrared-visible image fusion, reinforcement learning, chained semantic diagnosis, foundation model priors

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_n2yVOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript completes infrared and visible light image fusion based on pre-trained representations and calculable image quality rewards. The model retains the frozen visual basic model and visual language backbone, uses trainable branches to complete feature adaptation, and introduces perturbations in training while generating multiple fusion candidates, constructing a relative difference set, and using a combination of different rewards to complete the quality assessment of the fusion results; in the inference stage, only the reference candidates are output as the final fusion result. The overall experiments are rich and well-designed, and model training is completed using GRPO ideas based on similar reinforcement learning.

Critical strengths

This paper proposes a method that drives optimization based on the relative quality difference between candidates and reward evaluation, which is different from the traditional optimization paradigm for a single output under fixed loss. This method has certain novelties in the design of optimization goals and training signals. It can use the relative merits and demerits information between candidates and guide model optimization through reward evaluation. Experiments on multiple data sets show that this method has good effectiveness and shows certain practicality in downstream tasks. Overall, the paper has a clear idea, the method design is obviously different from the existing paradigm, and it has certain innovation and application value.

Critical weaknesses
  1. The connection and difference between CGPO and GRPO can still be further explained in order to understand the specific role of adjustments made for image fusion.
  2. There are differences in the performance of different reward combinations. How gradient rewards complement pixel and structure rewards requires further explanation.
  3. The role of semantic feedback and shared decoder in the intermediate stage can still be presented more intuitively, and a small amount of intermediate fusion results help to understand the step-by-step refinement process.
  4. The overall model has a large number of parameters. What is the current main computational bottleneck?
  5. What is the main basis for determining the number of candidate fusion results? What impact does the number of candidates have on model training cost and fusion quality?
Questions for authors
  1. What key ideas does CGPO draw from GRPO, and what adjustments has been made to image fusion?
  2. In Table 7, after adding gradient reward, the performance of the combination of pixel and structure reward is significantly improved. Does this result indicate that gradient information is particularly important for balancing different quality goals? What are the differences between the different reward packages?
  3. Compared with only introducing semantic guidance on the final fusion result, what problems are mainly solved by embedding feedback in the intermediate stage?
  4. The same decoder is used for intermediate image reconstruction and final candidate generation. What is the purpose of this design?
  5. It is recommended to supplement the fusion results of intermediate reconstruction to visually illustrate the changes in image quality during the step-by-step refinement process.
4 comments
AuthorsOpenReview ↗

Response to Reviewer n2yV (Part 1/2)

Thank you for your review and valuable suggestions for improving our method. We respond to your comments and questions below.

Q1 / W1: What ideas does CGPO draw from GRPO, and what adaptations are made for image fusion?

CGPO draws on GRPO’s core idea of optimizing through reward comparisons among multiple candidate outputs for the same input. It aims to extend single-output optimization to optimization based on relative quality comparisons among candidates. Specifically, CGPO uses features refined through chained semantic feedback to construct one unperturbed reference candidate and several Gaussian-perturbed candidates in feature space. These candidates are decoded into fused images by the shared decoder, and the model is optimized using the computed rewards. During training, multiple candidate fused images provide local quality references. The model computes relative advantages by subtracting the group-mean reward from each candidate’s reward and uses these advantages to update its parameters, guiding improvements in the quality of the reference fused image.


Q2 / W2: How does the gradient reward complement the pixel and structural rewards, and how do different reward combinations compare?

The pixel reward uses the pixelwise maximum of the source images as a reference and preserves strong brightness responses. The SSIM reward focuses on the main structures, while the gradient reward emphasizes edges and textures, providing direct feedback for preserving local details. Combining these three rewards helps balance brightness, structure, and detail.

Furthermore, the quantitative results in Table 7 show varying degrees of performance degradation when reward components are omitted, compared with using the full reward combination. The qualitative results in Figure 11 further illustrate the differences among reward combinations. Combining the pixel and SSIM rewards accounts for both brightness and structure, but textural details may remain unclear. Combining the pixel and gradient rewards preserves both brightness and detail but provides relatively weaker overall structural preservation. The SSIM and gradient combination retains richer details but produces darker images. When all reward components are included, the fused images achieve a better balance among brightness responses, structural preservation, and detail preservation. These results illustrate the overall benefits of jointly optimizing all three rewards.


Q3: What is the role of intermediate semantic feedback compared with guiding only the final fusion result?

Intermediate semantic feedback enables the model to adjust subsequent fusion features based on the current fusion state before generating the final fused image. Compared with applying semantic guidance only to the final result, we use diagnostic feedback from the current intermediate fused image to modulate the next-stage features within the same fusion process, with subsequent diagnosis based on the updated fused image. Thus, semantic feedback is continually updated as the fusion state evolves and, together with stage-specific prompts, progressively guides feature adjustments according to the semantic requirements of each stage.


Q4: Why is the decoder shared between intermediate image reconstruction and final candidate generation?

Sharing the decoder ensures that intermediate fused images and final candidate fused images are reconstructed through the same decoding mapping, providing a consistent basis for semantic diagnosis and candidate evaluation. If decoders with separate parameters were used, even with the same architecture, image differences could arise from both the fusion features and differences in the decoding mappings, potentially creating a mismatch between intermediate diagnostic feedback and the reconstruction characteristics of the final candidates. Furthermore, decoder parameters updated using the final candidate rewards are reused for subsequent intermediate fusion reconstructions, linking intermediate semantic feedback with the optimization of final fusion quality.

AuthorsOpenReview ↗

Response to Reviewer n2yV (Part 2/2)

Q5 / W3: Please provide intermediate fusion results to illustrate stage-wise changes in image quality.

To visualize changes in image quality during progressive refinement, we have added intermediate fusion visualizations to the revised manuscript (Appendix A.10.4, pp. 24–25). The newly added Figure 14 (p. 25) presents the intermediate images reconstructed by the shared decoder at Stages 3, 2, and 1, together with the final result at Stage 0.

Stage 3 reconstructs the initial fused image from features that have not undergone semantic modulation. Stage 2 uses feedback generated from the Stage 3 image and the prompt focusing on salient targets and global structure to modulate its features, increasing overall brightness and target responses. Stage 1 uses feedback from the Stage 2 image and the prompt focusing on boundaries and local contrast, reducing excessive brightness and producing clearer structural contours. Stage 0 uses feedback from the Stage 1 image and the prompt focusing on fine texture and artifact regulation, further reducing local brightness unevenness and producing a final image with more balanced brightness and more prominent pedestrians and vehicles.


W4: Why does the model have a large parameter count, and what is its main computational bottleneck?

As shown in Table 9, most parameters belong to the frozen DINOv3 and Qwen3-VL-Embedding backbones, while 17.2929 M parameters are trainable. For 256×256256\times256 inputs, Qwen3-VL-Embedding accounts for approximately 85% of the total FLOPs. The main bottleneck is thus semantic embedding extraction across multiple stages of chained diagnosis.

Under the evaluation conditions used in the paper, the complete inference pipeline takes an average of 0.3421 s per image pair on the MSRS test set at its original resolution. Our design reflects a trade-off between fusion quality and inference efficiency.


W5: How is the number of candidates selected, and how does it affect training cost and fusion quality?

We conducted a hyperparameter analysis with N∈1,2,3,4,5N \in \\{1,2,3,4,5\\} to evaluate the effect of candidate count on fusion quality. As shown in Figure 10, when N=1N=1, the candidate’s reward equals the group-mean reward, making the relative advantage identically zero. The objective therefore provides no effective training gradients, preventing the model from being effectively optimized through this objective.

When N=2N=2 the reference and perturbed candidates allow within-group quality comparisons, enabling optimization based on relative advantages. Increasing the candidate count provides more within-group quality references. The best performance among the tested settings is achieved at N=4N=4. However, increasing NN further to 5 reduces performance while increasing training cost, indicating that more candidates do not necessarily lead to continued improvements in fusion quality. An excessive number of candidates may make optimization more difficult without necessarily helping to improve the quality of the reference fused image. We therefore choose N=4N=4 as the final setting to balance fusion quality and training cost.

Increasing the candidate count mainly increases the training cost of image decoding and reward computation. This additional overhead occurs only during training. At inference, no perturbed candidates are generated, and the final fused image is reconstructed solely from the reference candidate.

The authors' response has addressed most of my concerns. After considering the other reviewers' comments and the authors' replies, I have decided to keep my original positive rating.

Thank you very much for your reply. Your comments have been very helpful to us. Thank you again for taking the time to review our manuscript.

Reviewer_7NvkOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies infrared-visible image fusion and introduces reward-based evaluation using relative differences among candidates, reframing conventional fusion loss minimization in terms of reward maximization. The proposed framework comprises three components. First, a frozen DINOv3-ConvNeXt provides hierarchical visual priors, while a spatial prior injection branch supplements local details. Second, a frozen Qwen3-VL-Embedding extracts stage-specific semantic embeddings from intermediate fused images, which progressively refine fusion features through visual-semantic alignment and channel modulation. Finally, a reference candidate and perturbed candidates are generated in feature space and optimized using a within-group comparison objective that combines pixel, structural, and gradient rewards.

Critical strengths
  1. The shift in the fusion paradigm is compelling. Using relative quality differences among candidates and reward-driven optimization offers a new perspective on conventional supervision with fixed loss functions.
  2. The experimental evaluation is extensive and reasonably comprehensive. The method performs well across multiple datasets, with downstream tasks and ablation studies providing further evidence of its effectiveness.
  3. The paper is clearly structured and well presented. The method description and experimental analysis are logically organized, making the approach easy to follow.
  4. Using foundation-model representations to provide feedback on intermediate fusion states is an appealing idea. Progressively modulating fusion features through semantic feedback provides a valuable exploration of how foundation models can contribute to the fusion process.
Critical weaknesses
  1. The specific distinction between semantic diagnosis and general semantic guidance remains unclear.
  2. Semantic modulation and candidate optimization appear complementary, but the explanation of why they work well together could be strengthened.
  3. The stage-specific prompt design is reasonable, and a simple controlled comparison could further substantiate its role.
  4. The object detection results show a small improvement, but the stability of this gain requires further clarification.
Questions for authors
  1. How does semantic diagnosis differ from general semantic feature guidance or modulation?
  2. How does channel modulation help mitigate the weakening of local targets or local artifacts?
  3. What are the main advantages of within-group relative reward optimization over directly maximizing the reward of the reference candidate?
  4. In Table 3, the benefits of CGPO depend on semantic modulation. How do the authors explain this observation?
  5. What principles guide the design of the stage-specific prompts, and how do these prompts address the fusion needs at each stage?
  6. The object detection results slightly outperform the strongest baseline. Could the authors provide further evidence or discussion regarding the stability of this small improvement?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 7Nvk (Part 1/2)

Thank you for taking the time to provide valuable feedback on our manuscript. We address your comments point by point below.

Q1 / W1: How does semantic diagnosis differ from conventional semantic feature guidance or modulation?

When source-image semantic features or text instructions are used as fixed guidance conditions, these conditions remain unchanged throughout fusion. Our semantic diagnosis instead generates feedback from the current intermediate fused image and stage-specific prompts to dynamically modulate the next-stage features, allowing semantic guidance to evolve with the fusion state. At each diagnostic stage, the reconstructed fused image is used to generate feedback, forming a continuous sequence of intermediate reconstruction, semantic feedback, and next-stage modulation.


Q2: How does channel modulation help mitigate local target attenuation or local artifacts?

To improve local details and suppress artifacts, we design a stage-specific prompt focusing on fine texture and artifacts for Stage 1. This prompt is combined with the current fused image to generate semantic feedback, which modulates the Stage 0 features to produce the final fused image. Specifically, we have added intermediate fusion results to the revised manuscript (Figure 14, p. 25). From Stage 1 to Stage 0, artifacts such as local brightness unevenness are reduced, while pedestrians and vehicles become more prominent. These visual results illustrate the role of the channel modulation design in mitigating local target attenuation and suppressing artifacts.


Q3: What advantages does group-relative reward optimization offer over directly maximizing the reference candidate's reward?

If the reward of the reference candidate output is directly maximized, model training is essentially equivalent to conventional fusion training, namely optimizing a predefined objective for a single output.

In contrast, the proposed CGPO introduces local comparisons among multiple candidate outputs generated for the same input, allowing the model to be updated based on relative quality differences among candidate fused images and encouraging the reference output to achieve a higher reward than the perturbed outputs. This extends “single-output optimization” to “optimization based on relative quality comparisons among candidates,” thereby providing relative quality references from neighboring candidates for optimizing the reference output.

Furthermore, S6 in Table 3, which follows this “single-output optimization” paradigm, still underperforms the complete model overall. This further validates CGPO’s advantage in fusion quality over directly maximizing the reward of the reference candidate output.


Q4 / W2: Why do the benefits of CGPO depend on semantic modulation, and how do the two components work together?

CSD-PR supports CGPO by extracting semantic feedback from intermediate fusion states and progressively adjusting features, thereby providing CGPO with semantically refined fusion representations. Compared with features that have not been refined through such semantic feedback, these representations additionally incorporate stage-specific semantic diagnostic information, providing a more informative basis for candidate generation and relative quality comparison in CGPO.

Furthermore, as shown in Table 3, removing either CSD-PR or CGPO reduces model performance. The configuration with CGPO but without CSD-PR underperforms the configuration with CSD-PR but without CGPO on most metrics. When both components are used together, the model’s overall performance improves noticeably. Together, these results support the effectiveness of CSD-PR and CGPO and their synergistic interaction.

AuthorsOpenReview ↗

Response to Reviewer 7Nvk (Part 2/2)

Q5 / W3: What principles guide the design of stage-specific prompts, what role do they play, and what advantages do they offer over a shared prompt?

The stage-specific prompts follow a coarse-to-fine design. Stage 3 focuses on salient targets and global structure. Stage 2 emphasizes object boundaries, regional contrast, and structural continuity. Stage 1 focuses on local textures and artifact suppression. Each diagnostic stage generates semantic feedback from the current intermediate fused image and applies it to the next stage. Compared with sharing the same prompt across stages, stage-specific prompts can enhance fusion features at different levels according to the requirements of each stage, thereby improving the final fusion quality.

To evaluate the effectiveness of stage-specific prompts, we have added a comparison with a shared fixed prompt to the revised manuscript (Appendix A.10.2, pp. 23–24). The newly added results (Figure 12, p. 24) show that the model using stage-specific prompts achieves better overall performance than the model using a shared fixed prompt.


Q6 / W4: How can the stability of the small improvement in object detection be assessed?

To assess the stability of the detection gains across detectors, we have added evaluations using YOLOv8-m and YOLOv8-l alongside the original YOLOv8-s experiments, comparing our method with the strongest baseline, FreeFusion. The additional experiments use the default random seed and follow the training and testing procedures described in the manuscript.

The mAP50:95\mathrm{mAP}_{50:95} results for the three detectors are as follows:

DetectorOursFreeFusion
YOLOv8-s0.65070.6501
YOLOv8-m0.66570.6531
YOLOv8-l0.66170.6597
Mean0.65940.6543

Our method achieves higher mAP50:95\mathrm{mAP}_{50:95} than FreeFusion with all three detectors, showing a consistent performance advantage across the tested detection models.

Reviewer_KGmBOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on a worthy research topic: how to utilize the representation of pre-trained models while retaining the local details required for the fusion of infrared and visible light. The authors combine the frozen hierarchical visual encoder with the spatial prior injection branch, adjust the multi-layer fused features through semantic embedding, and optimize the output during training by comparing the rewards between perturbation candidates. The reward is calculated based on the pixel, structure, and gradient relationships between the source image and the candidate image. The overall scheme is verified through fusion metrics on three datasets, downstream visual tasks, and design ablation studies.

Critical strengths

(1) The method shifts from single-output optimization under fixed loss constraints to reward-driven optimization based on candidate quality differences, providing an attractive paradigm exploration for image fusion. (2) Training signals can be constructed without the need for true values or additional manual preference annotations, which is in line with the actual conditions where supervision information is difficult to obtain in image fusion tasks. Moreover, the combination of pre-trained visual representations and local detail compensation helps to balance the retention of overall structure and fine-grained information. (3) The chain-like semantic modulation incorporates intermediate reconstruction results into the step-by-step refinement process, enabling the fusion process to continuously adjust features using phased feedback. (4) The article structure is complete and the experiments are quite diverse.

Critical weaknesses

(1) During the training phase, multiple perturbation candidates are compared, while in the inference stage only the reference candidate is used. It still needs further explanation as to how the benefits are implemented in terms of the quality improvement of the reference candidate. (2) What is the basis for choosing a fixed standard deviation σ = 0.1 for the perturbation? Additionally, the relationship between this and the numerical scale of the fused features needs to be further clarified and explained. (3) Infrared and visible light images have different imaging characteristics. How can the shared frozen encoder balance the information requirements of the two modalities? (4) The specific impact of semantic feedback on the intermediate reconstruction results lacks an intuitive explanation. It is currently difficult to determine which image contents have been mainly improved in each stage. (5) How does the relative quality difference between candidates affect the model? Additionally, this design differs from GRPO and needs to be further clarified.

Questions for authors

please see my detailed comments above

4 comments
AuthorsOpenReview ↗

Response to Reviewer KGmB (Part 1/2)

Thank you for your careful review and the time and effort you devoted to providing valuable, constructive feedback on our manuscript. We respond to each point below.

W1: Training compares multiple perturbed candidates, whereas inference uses only the reference candidate. How do the benefits of multi-candidate training translate into improved reference candidate quality?

As shown in Eq. (12) , we combine all candidates' relative advantages with corresponding normalized candidate log densities to form the candidate optimization loss LRL\mathcal L_{\mathrm{RL}}. The reference candidate lies at the Gaussian mean and has the highest log density. The objective encourages a higher reward for the reference fused image. Instead, for perturbed candidates whose log densities are below the group-mean log density, it directs optimization toward reducing their image rewards. This encourages a larger reward gap between the reference fused image and these perturbed images, which guides the reference fused image toward higher quality as measured by the reward.


W2: Why is the perturbation standard deviation fixed at σ=0.1\sigma=0.1, and how does it relate to the numerical scale of the fusion features?

To assess the suitability of σ=0.1\sigma=0.1 as the default perturbation standard deviation, we have added a sensitivity analysis in the revision manuscript (Appendix A.10.3, pp. 23–24), evaluating σ∈1,0.1,0.01,0.001,0.0001,0.00001\sigma \in \\{1, 0.1, 0.01, 0.001, 0.0001, 0.00001\\}. As shown in the newly added Figure 13 (p. 24), σ=0.1\sigma=0.1 achieves the best performance.

σ\sigma is the standard deviation of the Gaussian noise added to the policy mean feature μ\mu. It specifies an absolute perturbation scale. When the perturbation is too small, differences between candidate feature, making it difficult to obtain effective optimization signals from quality comparisons. Instead, when the perturbation is too high, candidate features may deviate too far from the current fusion state, weakening local comparisons. We therefore adopt (σ=0.1\sigma=0.1 ) to balance candidate diversity with proximity to the current fusion state.


W3: How does the shared frozen encoder accommodate the information needs of both infrared and visible modalities?

We use a shared, frozen DINOv3-ConvNeXt encoder to extract multi-scale features from infrared and visible images separately, facilitating cross-modal feature association and complementary information fusion within a common feature space. To better meet the fusion needs of both modalities, we use SPIB to extract local spatial features from the channel-wise concatenated original infrared and visible images through a shallow convolutional network, providing supplementary information to address the shared encoder’s potential information needs during multi-scale feature extraction.

AuthorsOpenReview ↗

Response to Reviewer KGmB (Part 2/2)

W4: How does semantic feedback affect intermediate reconstructions, and what image content does each stage primarily improve?

To illustrate how semantic feedback affects intermediate fusion states, we have added intermediate fusion visualizations to the revised manuscript (Appendix A.10.4, pp. 24–25). The newly added Figure 14 (p. 25) shows the intermediate images reconstructed by the shared decoder at Stages 3, 2, and 1, together with the final result at Stage 0.

As shown in Figure 14, Stage 3 reconstructs an initial fused image from fusion features that have not undergone semantic modulation. Stage 2 receives semantic feedback generated from the Stage 3 fused image and a prompt focusing on salient targets and global structure. After feature modulation, the reconstructed image exhibits increased overall brightness and enhanced target responses. Stage 1 receives feedback generated from the Stage 2 fused image and a prompt focusing on boundaries and local contrast, resulting in reduced excessive brightness and clearer structural contours. Stage 0 then uses feedback generated from the Stage 1 fused image and a prompt focusing on fine texture and artifact regulation to further reduce local brightness unevenness, producing a final fused image with a more balanced brightness distribution and more prominent pedestrians and vehicles. These results show that chained semantic feedback can combine the current fusion state with stage-specific prompts to progressively modulate subsequent features and improve fusion results.


W5: How do relative quality differences between candidates affect the model? How is CGPO related to GRPO, and how do they differ?

Quality differences between candidates are converted into relative advantages by subtracting the group-mean reward from each candidate’s reward. These advantages, together with the corresponding normalized log densities, form the optimization objective. During training, gradients of this objective are backpropagated through reward computation and candidate image reconstruction to update all trainable network parameters.

GRPO samples multiple candidate outputs for the same input, estimates relative advantages from within-group rewards, and uses these advantages to weight policy updates. CGPO draws on this within-group comparison principle and makes task-specific adaptations for image fusion. Specifically, CGPO constructs unperturbed reference candidate features and perturbed candidate features from fusion features refined through chained semantic feedback. These features are decoded into multiple candidate fused images by the shared decoder, and the source images are used to compute the composite reward for each candidate fused image. Relative advantages are obtained by subtracting the group-mean reward from each candidate’s reward. Unlike GRPO’s policy update, CGPO backpropagates gradients through candidate rewards and image reconstruction to update the trainable network parameters, allowing relative quality differences between candidates to guide improvements in the quality of the reference fused image.

thx to the authors for their detailed responses. These responses have addressed all of my concerns, so I have decided to raise my score.

We learned a great deal from your thoughtful comments. Thank you very much for your reply and decision, which mean a lot to us. We sincerely appreciate the time you took to review our manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.