acceptodds
Under review as a conference paper at ICLR 2027

TriReward-Fusion: Unified Vision-Language Reward Learning for Task-Oriented Infrared–Visible Image Fusion

Abstract

Infrared and visible image fusion aims to combine the thermal saliency of infrared images with the structural details of visible images for robust high-level visual perception. However, most existing fusion methods are optimized using handcrafted no-reference pixel-level objectives or supervised by a single downstream task, making it difficult to jointly preserve visual fidelity, semantic integrity, and object-level reliability. Moreover, image fusion, semantic segmentation, and object detection pursue distinct and sometimes conflicting objectives, and a simple weighted combination of their losses often fails to provide unified and semantically meaningful feedback. To address this problem, we propose TriReward-Fusion, a BLIP-guided vision-language reward learning framework for task-oriented infrared-visible image fusion. The proposed framework is trained on MSRS by jointly using image-pair no-reference fusion objectives, semantic segmentation annotations, and our additionally annotated detection labels. It projects fusion, segmentation, and detection outputs into a shared vision-language semantic state space, where perceptual information preservation, semantic region completeness, and instance-level detection reliability are transformed into learnable multi-task reward signals. These rewards provide unified semantic feedback to guide the fusion process and reconcile pixel-level reconstruction, region-level parsing, and object-level localization objectives. Experiments evaluate fusion quality on MSRS, M3FD, and LLVIP, semantic segmentation on MSRS, and object detection on M3FD. The results show that TriReward-Fusion improves fused image quality and enhances downstream segmentation and detection performance, demonstrating the effectiveness of vision-language reward learning for unified multi-task infrared-visible image fusion.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Infrared-visible Image Fusion, Vision-Language Models, Multi-Task Optimization, Semantic Segmentation, Object Detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZKYZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TriReward-Fusion, a framework for task-oriented infrared–visible image fusion (IVF) that jointly optimizes three tasks: image fusion, semantic segmentation, and object detection. The central idea is to project the fused image and the predictions of the two downstream tasks into a shared BLIP-based vision–language “semantic state space,” and from it derive three learnable reward signals — perceptual preservation (Rf), semantic completeness (Rs), and detection reliability (Rd). The reward heads are first calibrated (with the task network frozen) against task-derived targets: a ranking loss between a fused image and a synthetically degraded version for Rf; a soft-IoU plus boundary-F target for Rs; and a bipartite-matched detection-quality target for Rd, together with a vision–language alignment loss to BLIP category text prototypes. In a second stage the calibrated reward model is frozen and used as a differentiable critic that feeds reward-guidance gradients back to the fusion network, using adaptive softmax reward-weight prediction and reward-conditioned feature modulation. A segmentation–detection consistency loss (soft box–mask agreement plus a Jensen-Shannon term on category presence) is added. The authors additionally annotate person/bicycle/car bounding boxes on MSRS to enable three-task training. Validation covers fusion quality on MSRS, M3FD, and LLVIP, segmentation on MSRS, and detection on M3FD, against twelve recent fusion methods, plus component ablations.

Critical strengths
  1. Unified, well-motivated formulation with a sensible staged-critic design. The three-task framing (fusion + segmentation + detection) sharing a vision–language semantic space is a reasonable direction, and the two-stage calibration — fitting reward heads to task-derived targets while the task network is frozen, then freezing the reward model before using it as a critic — shows explicit awareness of the co-adaptation / reward-hacking risk that plagues learned-reward training.
  2. Comparatively broad evaluation. Three fusion datasets, downstream segmentation and detection, twelve fusion baselines, component ablations, and a cross-dataset protocol (train on MSRS only; evaluate on M3FD and LLVIP) together make a stronger generalization case than is typical in this literature.
Critical weaknesses
  1. Limited novelty and unclear delta over the closest prior work. The core ingredients — VLM/text-guided fusion (Text-IF, Text-DiFuse, OmniFuse, IF-FILM, SAGE-SAM), learnable fusion objectives, and multi-task fusion-in-the-loop training — are all established. The closest work, Bai et al., “Task-driven image fusion with learnable fusion loss” (CVPR 2025), is cited but never compared or experimentally separated, even though it embodies essentially the same idea (learn a fusion objective/critic to drive the fusion network). The paper does not articulate a crisp technical delta. Relatedly, the “reward learning” framing is misleading: there is no reinforcement learning here — no policy, no sequential decision process, no return — only a supervised learned critic used as a differentiable auxiliary loss, which inflates the apparent contribution.
  2. Strength of the empirical evidence does not support the claims. (a) Table 1 SSIM values ≥ 1.00 are impossible for standard SSIM and must be corrected, which casts doubt on the metric pipeline. (b) The claimed gains are small and untested — e.g., downstream mIoU 76.32 vs. 74.58 (OmniFuse) and mAP50 84.11 vs. 82.85 — with no multiple runs, error bars, or significance tests. (c) The fusion-quality improvement rests on no-reference metrics that are easily gamed, an awkward basis for a method whose selling point is avoiding reward hacking. (d) The new MSRS detection annotations are used only for training while detection is evaluated on M3FD, so the value and quality of that annotation effort (no inter-annotator agreement reported) are unclear. (e) Reproducing twelve baselines, including diffusion-based ones, on a single RTX 2080 Ti is a strong claim that cannot be verified without released code/configs.
Questions for authors
  1. How were the SSIM values obtained such that several exceed 1.00 (IGNet 1.01/1.02, PIAFusion 1.00)? Please give the exact definition and implementation used for SSIM (and for MI and VIF) and reconcile the values with standard bounds.
  2. How does TriReward-Fusion differ technically and empirically from Bai et al. (2025, “Task-driven image fusion with learnable fusion loss”)? Please provide a direct comparison.
  3. Because the reward heads are calibrated against targets derived from the task network’s outputs and then frozen, as the network improves during stage 2 the critic may become stale. How do you ensure the frozen critic remains a valid quality estimator? Please provide a “reward vs. true quality” curve over training.
  4. Rf is trained only by relative ranking against synthetic degradations of the model’s own output. What evidence shows Rf tracks an external notion of fusion quality rather than simply preferring outputs close to the initial ones?
  5. Please report the full metric set for the ablation, add an ablation that replaces BLIP with a non-vision-language feature (to isolate the VLM contribution), and add an ablation of the two-stage calibration vs. joint training.
  6. Are the reported improvements statistically significant? Please report results over multiple seeds/runs.
  7. Why is detection evaluated only on M3FD when the new annotations are on MSRS? If the detection reward is trained on MSRS categories (person/bicycle/car), how does it transfer to M3FD’s target set?
  8. What is the added training and inference cost of BLIP plus the reward heads plus two-stage training, relative to the “w/o Tri-reward” variant and to the baselines?
Reviewer_vMeUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces TriReward-Fusion, a unified framework for fusing infrared and visible images which combines fusion, semantic segmentation and object detection by means of a BLIP-guided vision-language reward model. The main concept consists of projecting the outputs of the three tasks into a common semantic state space and learning three rewards that are specific to each task—perceptual preservation, semantic region completeness and instance-level detection reliability—these rewards being calibrated using task supervision and then fixed to assist the fusion process. The framework also makes use of adaptive reward aggregation, reward-conditioned feature modulation, and a segmentation–detection consistency loss, resulting in better performance according to standard fusion metrics and in terms of downstream segmentation and detection performance on MSRS, M3FD and LLVIP.

Critical strengths
  • The use of a VLM (BLIP) to create a shared semantic state in the process of fusion, segmentation, and detection represents a well-considered and far from simple extension of previous VLM-guided fusion research.
  • The use of task-derived targets for reward calibration (this including pairwise ranking when fusing the reward with targeted degradations) is well justified and extends beyond simply using the raw VLM scores.
  • The combination of adaptive reward aggregation, reward-conditioned feature modulation (affine feature scaling) and cross-task consistency is cohesive and addresses multi-task conflicts from different levels
Critical weaknesses
  • The fact that BLIP (which was trained on natural images) is used means that there is a domain gap concerning IR–VIS imagery; even though this gap is to some extent reduced by calibration and alignment, there is still a lack of detailed analysis of the failure modes and sensitivity to the choice of prompt.
  • The reward learning makes use of the segmentation/detection ground truth from MSRS, and the generalization of the learned rewards to datasets having different distributions or categories is not extensively examined.
  • The assumption of a shared category (segmentation/detection) reduces the semantic range of reward guidance; it is not clear how the method can be extended to more comprehensive taxonomies or to scenarios with an open vocabulary.
Questions for authors
  • To what extent does performance depend on the selection and number of text prompts in each category? Can you give an ablation study involving prompt ensembles and the temperatures (τ_v, τ_a)?
  • What happens to the performance of the tri-reward when the segmentation and detection categories go beyond the three common classes or vary from one dataset to another? Is the framework able to handle open-vocabulary categories or zero-shot classes (for example, by using text-only prototypes without any labels)?
  • What would happen to TriReward-Fusion if it were trained without segmentation or detection supervision (that is, only using the calibrated fusion reward) or if it were trained with segmentation supervision alone? This approach would allow us to separate out the value of each kind of supervision from that of the VLM rewards.
Reviewer_quPJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new vision-language reward learning framework for infrared-visible image fusion. The proposed method jointly considers image fusion, semantic segmentation, and object detection, in which the BLIP is used to construct a shared semantic space and learn three task-specific rewards. Then, these rewards are combined through adaptive weighting and feature modulation to guide fusion optimization. The provided experiments demonstrate that the proposed method improves the fusion quality and downstream perception performance.

Critical strengths
  • This paper explores the integration of image fusion, semantic segmentation, and object detection within a unified framework, which is important for enhancing the practical applicability of image fusion.

  • The proposed BLIP-guided reward mechanism effectively balances perceptual fidelity, semantic completeness, and detection reliability.

  • The proposed method achieves improvements in both low-level fusion metrics and high-level perception performance, indicating that the learned fusion representations can benefit multiple visual tasks.

Critical weaknesses
  • The three rewards are learned from handcrafted fusion preferences and conventional task metrics, while the original task losses are still retained during training. It is unclear what advantages these learned rewards offer over directly optimizing the corresponding losses, particularly in addressing the claimed gradient conflicts.

  • The necessity of introducing BLIP deserves further justification. Comparisons with CNN- or ViT-based reward models would help distinguish the benefits of vision-language semantics from those of additional supervision and model capacity.

  • Although the three reward heads share a semantic feature space, they are trained independently using different task-specific targets. Whether this design enables meaningful collaboration among the three tasks remains questionable.

  • The experiments lack separate fusion+segmentation and fusion+detection configurations, making it difficult to assess whether the two tasks provide complementary benefits within the unified framework.

  • Considering the multiple task networks, BLIP encoders, and reward modules, an analysis of computational complexity, and inference speed is needed to evaluate the practicality of the proposed framework.

  • The visual improvements in Fig. 2 are not sufficiently distinguishable from existing methods. More representative qualitative comparisons would better support the claimed advantages.

Questions for authors

Please see the weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.