acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Task-Oriented Image Fusion as Probabilistic Inference

Abstract

Although task-oriented image fusion has achieved promising downstream performance by incorporating task supervision, existing methods typically learn a direct mapping from multimodal inputs to a single fused image, with task objectives serving mainly as optimization signals. Such a formulation leaves the underlying conditional distribution of task-relevant fusion results unmodeled, making it difficult to capture the distributional structure and statistical regularities of multimodal integration for downstream tasks. To address this limitation, we rethink task-oriented image fusion as probabilistic inference, modeling the fused image as a conditional distribution jointly determined by source modalities and downstream task information. We introduce a latent fusion variable that captures both multimodal fusion cues and task-relevant semantics, which is sampled and decoded to generate fused images. During training, task annotations are used to construct a task-informed posterior, whose knowledge is transferred to a source-conditioned prior for label-free inference. We further align source modal and fused image task predictions to preserve task-relevant information. Experiments on infrared-visible fusion with semantic segmentation and object detection demonstrate consistent improvements in both fusion quality and downstream performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Fusion, Task-Oriented Fusion, Probabilistic Inference, Downstream Perception, Variational Learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_MdAuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a probabilistic framework for task-oriented image fusion. The core idea is to model fusion as a conditional distribution guided by source modalities and downstream task information. A latent-variable framework based on variational inference is introduced to transfer task knowledge from a task-informed posterior to a source-conditioned prior, enabling label-free inference. A task consistency constraint further encourages the preservation of task-relevant information. Experiments on two datasets demonstrate competitive fusion quality and downstream task performance.

Critical strengths
  • Formulating task-oriented image fusion from a probabilistic perspective provides is an interesting and practical direction.
  • The design of posterior-prior framework is reasonable.
  • The experiment is relatively comprehensive, covering both image fusion quality and downstream perception tasks, including object detection and semantic segmentation. Component-level ablations and visualizations are also provided.
Critical weaknesses
  • The connection between Eq. (5) and the latent variable modeling and training objective is not clear enough.
  • The proposed posterior-prior framework is closely related to conditional variational inference and knowledge distillation, the key technical differences from existing probabilistic learning frameworks should be discussed.
  • The ELBO derivation involves a likelihood term for the fused image, while the practical training objective relies on intensity and gradient preservation losses without an explicitly defined likelihood model. A clearer explanation of their relationship is required.
  • The ablation results in Figure 6 show that removing certain components improves several fusion metrics, which is not fully consistent with the discussion. Also, the optimization objective in Algorithm 1 differs from Eq. (22).
  • Some minor typos: (1) In Figure 1, the visible modality is incorrectly labeled as "Infrared B" instead of "Visible B". (2) The reparameterization formula in Figure 2 uses the task-informed posterior during inference.
Questions for authors

Please refer to the Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer MdAu

Thank you for your constructive comments and valuable suggestions.

Response to W1: Specifically, Eq. (5) establishes the target task-conditioned fusion distribution as the combination of a source-dependent fusion prior and a task likelihood. To make this distribution tractable, we introduce the latent variable Z in Section 3.2 and decompose the fusion process into latent inference and image generation (Eq. (10)). In Section 3.3, the task-informed posterior qϕ(Z∣A,B,Y)q_\phi(Z|A,B,Y) is introduced during training to approximate the intractable latent posterior, while the source-conditioned prior pθ(Z∣A,B)p_\theta(Z|A,B) is learned through KL divergence alignment for label-free inference. The resulting variational objective (Eq. (22)) consists of the fusion constraint loss and posterior-prior alignment loss. Finally, Section 3.6 incorporates these terms into the overall objective together with task supervision and task consistency constraints. In particular, the fused-image task supervision provides the practical counterpart of the task likelihood in Eq. (5).

Response to W2: Thank you for this insightful comment. We acknowledge that our framework is related to conditional variational inference and knowledge transfer. However, our method differs in its task-oriented formulation and inference mechanism. Unlike conventional conditional variational frameworks, our posterior$q_\phi(Z|A,B,Y)$ incorporates task annotations during training, while the prior pθ(Z∣A,B)p_\theta(Z|A,B) relies only on source modalities, enabling task-free inference through posterior-prior alignment. Moreover, unlike conventional knowledge distillation that mainly transfers knowledge through feature or prediction matching, our framework transfers task-related knowledge through latent distribution alignment, complemented by task supervision and prediction consistency constraints. This combination enables task-aware probabilistic fusion without requiring task annotations during inference. We have clarified these connections and differences in the revised manuscript.

Response to W3: Thank you for raising this important concern. We agree that the relationship between the variational likelihood and the practical fusion objective requires further clarification. In our framework, since ground-truth fused images are unavailable, we do not directly define a reconstruction likelihood over the fused image. Instead, we introduce source-derived fusion constraints$\mathbf C_{\mathbf A,\mathbf B}$ and formulate a constraint-based likelihood to evaluate whether the generated fused image satisfies these constraints (Eqs. 16–17). Specifically, the constraints are implemented through intensity and gradient preservation terms, whose negative log-likelihood is explicitly connected to the practical fusion loss in Eq. (21). Based on this formulation, the variational objective in Eq. (22) consists of the expected fusion constraint loss and the KL divergence between the task-informed posterior and the source-conditioned prior. Therefore, the practical fusion loss is interpreted as the negative log-likelihood of satisfying source-derived fusion constraints, rather than reconstructing a predefined ground-truth fused image. We have clarified this relationship in the revised manuscript to make the probabilistic formulation and its practical implementation more consistent and understandable.

Response to W4: We agree that the original discussion did not sufficiently reflect the metric-dependent trade-offs in the ablation results. We have revised Section 4.6 to acknowledge that the full model does not consistently outperform all ablated variants across individual fusion metrics. Instead, we clarify that different components contribute complementary properties, with the full model achieving stronger overall downstream task performance.

Regarding Algorithm 1, we have revised the algorithm to correct the inconsistency in the original optimization procedure and ensure that it accurately reflects the overall training objective presented in Section 3.6.

Response to W5: Thank you for carefully identifying these errors. We have corrected both issues in the revised manuscript. Specifically, in Figure. 1, "Infrared B" has been corrected to "Visible B". In Figure. 2, the inference-stage latent representation has been corrected to use the source-conditioned prior mean μp\boldsymbol{\mu}_p instead of the task-informed posterior parameters. These corrections ensure consistency with the actual inference procedure.

Thank the authors for their replies. All of my concerns are resolved. I do not have further questions.

AuthorsOpenReview ↗

Reply to Reviewer MdAu

We greatly appreciate your kind recognition. Thanks again for your professional comments.

Reviewer_qYTsOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

A probabilistic framework is proposed for task-oriented infrared and visible image fusion. It introduces a latent variable to model the fusion process and uses a task-informed posterior aligned with a source-conditioned prior to transfer task information learned during training and enable inference without task labels. A task consistency loss is also introduced to preserve task-relevant information. Experiments on the M3FD and FMB datasets show competitive fusion performance and improvements in some downstream task metrics compared with existing methods.

Critical strengths
  1. As mentioned in the paper, probabilistic modeling has not been widely explored in the field of image fusion. Using a mathematical framework with a solid theoretical foundation to design fusion methods also opens up new research directions for related approaches.

  2. The paper is well-written and clearly organized. The overall logic is easy to follow, and the figures and tables are presented clearly, making the proposed method and experimental results easy to understand.

Critical weaknesses
  1. The theoretical formulation relies on assumptions that are not sufficiently justified. In Section 3.1, the simplification from P(Y|F,A,B) to P(Y|F) assumes that the fused image preserves sufficient information for the downstream task. However, some task-relevant information from the source modalities may be lost during fusion.

  2. The implementation of the proposed probabilistic framework is not sufficiently explained. In Figure 2, the transformation from the latent variable Z to the fused image is implemented through a neural network mapping. However, it remains unclear whether this mapping can adequately model the claimed distribution transformation and generation uncertainty. The latent representation may also affect the preservation of source information and the stability of fusion results. Further analysis of the latent representation and the role of stochastic sampling would help validate the effectiveness of the proposed method.

Questions for authors

1.The relationships among the three subfigures in Figure 1 are not clearly explained, and the description is limited.

2.In fact, generative frameworks such as GANs and diffusion models are essentially forms of probabilistic modeling and distribution transformation. The authors could try to explain the connections and differences between them in the paper to better clarify the motivation of this work.

  1. In Section 3.1, the derivation of Eq. (5) relies on the assumption that the fused image preserves sufficient information for the downstream task, allowing P(Y|F,A,B) to be simplified to P(Y|F). However, this assumption is not sufficiently justified. In image fusion, some task-relevant information from the source modalities may be lost during fusion.

  2. The latent variable Z involves changes in channel dimensions and distribution transformations. Could this lead to information loss and even affect the subsequent fusion results?

  3. In Figure 2, the distribution transformation from the latent variable to the fused image is actually implemented as a network mapping process, which is relatively simple. Can this implementation adequately fulfill the role of distribution transformation?

  4. The response maps in Figure 7 clearly highlight important target regions such as pedestrians. However, the mechanism by which the incorporation of downstream task information affects these responses is not clearly explained, making it difficult to fully understand what they represent.

3 comments
AuthorsOpenReview ↗

Response to Reviewer qYTs

Thank you for your constructive comments and valuable suggestions.

Response to Q1: Figure 1 illustrates the progression from conventional fusion to task-driven fusion and our method: (a) conventional fusion generates the fused image from source modalities, focusing on visual information preservation without task guidance; (b) task-driven fusion introduces downstream task supervision to guide fusion optimization; (c) our method explicitly models fusion as a task-conditioned distribution, using task annotations to form the posterior during training and a source-conditioned prior for label-free inference, with task consistency aligning source and fused-image predictions. We have clarified these relationships in the revised text and figure caption.

Response to Q2: We agree that GANs and diffusion models are also generative approaches for modeling image distributions. However, GANs typically rely on adversarial distribution matching, while diffusion models learn image distributions through iterative denoising. In contrast, our method explicitly models task-conditioned latent fusion through a variational posterior-prior framework, transferring task knowledge from a task-informed posterior during training to a source-conditioned prior for label-free inference. We have clarified these connections and differences in the Related Work section.

Response to Q3: We agree that some task-relevant information may inevitably be lost during image fusion. Our assumption does not imply that the fused image preserves all information from the source modalities, but rather that it retains sufficient information for the target task. This is a task-sufficiency approximation adopted to simplify the probabilistic formulation. In our framework, fused-image task supervision and the task consistency constraint in Eq. (25) encourage the preservation of task-relevant information. Nevertheless, we acknowledge that this assumption does not guarantee complete preservation of all task-related information.

Response to Q4: Latent representation learning is a widely adopted strategy in computer vision, enabling compact representations of high-dimensional features. We acknowledge that dimensionality reduction may inevitably discard some information. However, it also reduces the computational burden of probabilistic modeling and facilitates efficient inference. The key is to preserve essential information relevant to the target task rather than all redundant source information. Our experimental results demonstrate that the learned latent representations retain sufficient information to achieve effective fusion and competitive downstream performance. Therefore, we consider this a reasonable trade-off between representation compactness, computational efficiency, and task performance.

Response to Q5: We clarify that the probabilistic modeling in our framework is primarily performed in the latent space, rather than through a stochastic decoder. Specifically, the latent variable is sampled from the learned posterior or prior distribution, and the decoder performs a deterministic nonlinear mapping from each latent sample to a fused image. Therefore, different latent samples can produce different fused images, inducing a distribution over fusion results. This design separates probabilistic representation learning from image generation, avoiding the need for a more complicated stochastic decoder. Since our objective is to capture task-oriented fusion uncertainty rather than model an arbitrary image distribution, we consider this latent-space probabilistic modeling combined with a deterministic decoder sufficient for our purpose.

Response to Q6: We clarify that the maps in Figure. 7 represent the spatial uncertainty of the learned latent distribution, rather than direct activation maps of the downstream task network. During training, task annotations are encoded and incorporated into the posterior network, while task supervision and consistency losses further guide the latent representations to preserve task-relevant information. These constraints influence the learned posterior distribution through backpropagation. Through posterior-prior alignment, the task-related guidance is transferred to the source-conditioned prior, allowing the prior to capture task-informed uncertainty without requiring task annotations during inference. Consequently, the highlighted regions, such as pedestrians and vehicles, indicate higher uncertainty in the latent fusion representations at these locations, rather than direct task-network responses.

Reviewer_qYTsOpenReview ↗

Rethinking Task-Oriented Image Fusion as Probabilistic Inference

The authors have well addressed all my concerns, and I will therefore raise my rating.

AuthorsOpenReview ↗

Reply to Reviewer qYTs

We greatly appreciate your kind recognition. Thanks again for your professional comments.

Reviewer_M4AtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper models task-oriented infrared–visible light fusion using latent variables, learning the posterior distribution during training via task annotations while training a prior that relies solely on source images to enable inference without task annotations. The method also aligns task predictions between source and fused images to preserve information required for downstream tasks. Detection and segmentation experiments report improvements on several fusion and task metrics, along with component ablation and visualizations of latent variable variance.

Critical strengths
  1. The posterior uses task annotations during training, while the prior requires only the source images during inference, providing a concrete mechanism for transferring task information to the fused representation. The consistency in task predictions between the source and fused images also provides a direct training signal for preserving information relevant to downstream tasks.
  2. Evaluations covered detection and segmentation, and the method achieved improvements on several metrics, including the highest detection precision on M3FD and mACC on FMB among the compared methods, as well as improvements on some fusion metrics. These results support that the overall training scheme can generate useful task-oriented representations, though the benefits are not consistent across all metrics.
Critical weaknesses
  1. Equations (16)–(20) introduce the likelihood of the fused image FF, but the training data described in the paper consists only of source images and task annotations. The paper does not specify what FF is as the observed target, nor does it define the specific form of p(F∣Z)p(F\mid Z). The actual objective in Equation (22) combines the fusion loss, task supervision, and distribution alignment, yet the correspondence between these components and the likelihood is not explained. Consequently, it is currently difficult to determine whether the training optimizes the probabilistic model proposed in the paper or primarily uses the posterior–prior KL alignment as regularization. This issue pertains to core modeling rather than a simple omission of notation.
  2. The benefits of the probabilistic component itself remain unclear. The specific implementation of the “w/o Prob.” ablation is not explained, making it impossible to determine whether the model capacity and supervision are comparable; it is also unclear whether the main experiment used the prior mean or random sampling. The interpretation of the ablation results remains too broad: on FMB, the CE of the full model is 2.216, which is worse than the 1.757–1.882 observed in all ablation versions; MI and SD are also lower, although AG and EI are better. This reflects trade-offs among the metrics. The variance heatmap provides a qualitative illustration but does not yet explain what independent benefits distribution modeling brings.
Questions for authors
  1. What is the fused image observation target FF in Equations (16)–(20)? How is p(F∣Z)p(F\mid Z) defined, and how does it correspond to the loss in Equation (22)? Algorithm 1 does not explicitly list the task supervision loss or the stop-gradient operation; is this merely an abbreviation, or does the actual implementation still include these steps?
  2. What specific alternative structure is used in “w/o Prob.”? Does it maintain comparable model capacity and the same task supervision? Do the main results use a prior mean or random sampling? Comparing with deterministic models under consistent conditions would help illustrate what probabilistic modeling adds.
  3. In Figure 6(b), the complete model has a higher CE and lower MI and SD, while AG and EI are better. How do the authors explain this trade-off? Should the statement in Section 4.6—that removing any component reduces fusion quality—be narrowed accordingly?
  4. What data partitions and downstream task networks were used in Figure 4? Were the task networks retrained separately under the same settings for the fused images of each method? Please clarify the task supervision obtained by each fusion method and the full training setup.
3 comments
AuthorsOpenReview ↗

Response to Reviewer M4At

Thank you for your constructive comments and valuable suggestions.

Response to W1 and Q1:

(1) Probabilistic interpretation and likelihood formulation.

Our framework models uncertainty through the latent fusion variable rather than stochasticity in the decoder. Specifically, the decoder implements a deterministic mapping F^=Dψ(Z)\hat{\mathbf F}=D_\psi(\mathbf Z), while Z\mathbf Z is sampled from the task-informed posterior during training or the source-conditioned prior during inference. Thus, different latent samples can produce different fused images, inducing a distribution:

=\int\delta(\mathbf F-D_\psi(\mathbf Z)) q_\phi(\mathbf Z|\mathbf A,\mathbf B,\mathbf Y)d\mathbf Z.$$ Regarding the likelihood formulation, since ground-truth fused images are unavailable, we introduce source-derived fusion constraints $\mathbf C_{\mathbf A,\mathbf B}$ instead of directly defining a reconstruction likelihood over $\mathbf F$. Specifically, the intensity and gradient residuals are modeled using factorized Laplace distributions. Their negative log-likelihoods correspond to the practical intensity and gradient preservation losses, respectively. Therefore, the constraint-based likelihood $p_\psi(\mathbf C_{\mathbf A,\mathbf B}|\mathbf Z,\mathbf A,\mathbf B)$ provides a probabilistic interpretation of the fusion objective. Based on this formulation, the negative ELBO leads to the variational objective: $$\mathcal L_{\mathrm{var}}=\mathbb E_{q_\phi} [\mathcal L_{\mathrm{fus}}(D_\psi(\mathbf Z);\mathbf A,\mathbf B)] +D_{\mathrm{KL}}( q_\phi(\mathbf Z|\mathbf A,\mathbf B,\mathbf Y) \|p_\theta(\mathbf Z|\mathbf A,\mathbf B)).$$ The fusion term preserves complementary source information, while the KL term aligns the task-informed posterior with the source-conditioned prior, enabling task-related knowledge transfer to label-free inference. The task supervision and consistency losses are further incorporated into the overall objective in Eq. (22). We have revised the manuscript to clarify the constraint-based likelihood formulation, its relationship with the practical fusion loss, and the probabilistic interpretation of the deterministic decoder. **(2) Clarification of Algorithm 1.** Both source-modality and fused-image task supervision losses, corresponding to $\mathcal L_{\mathrm{stask}}$ and $\mathcal L_{\mathrm{ftask}}$, are included in our implementation. For task consistency, stop-gradient is applied to the source-modality logits, treating them as fixed references while updating the fused-image branch. The source branch is independently optimized through its task supervision loss. These details were omitted from the original algorithm for brevity. We have revised Algorithm 1 to explicitly include both task supervision losses and the stop-gradient operation. **Response to W2 and Q2:** The w/o Prob. variant directly connects the cross-modal encoder to the fusion decoder, removing the Gaussian prior and posterior heads, label encoder, and reparameterization operation. The source encoders, cross-modal encoder, fusion decoder, and task prediction branches remain unchanged. Thus, the shared fusion and task network capacities are maintained, with parameter reduction limited to the removed probabilistic components. Both source-modality and fused-image predictions use the same ground-truth annotations and task losses as the full model, ensuring consistent task supervision. During training, latent variables are sampled from the task-informed posterior via reparameterization. During inference, the prior mean is used to ensure deterministic and reproducible results. Stochastic prior sampling is not used in the main experiments. **Response to W2 and Q3:** We agree that the FMB results reveal metric-dependent trade-offs. Specifically, the full model achieves the best AG (4.115) and EI (43.318), but has higher CE (2.216) and lower MI (2.205) and SD (9.267) than the ablated variants. This is because these metrics capture different properties that may not be simultaneously optimized. Our method prioritizes task-relevant information preservation and downstream performance rather than optimizing every fusion metric independently. We have therefore revised Section 4.6 to acknowledge these trade-offs and avoid overstating the improvements across all fusion metrics. **Response to W2 and Q4:** Figure 4 uses fused images from different methods on M3FD and FMB, with identical data partitions and fusion protocols. YOLOv10 and SegFormer are used for detection and segmentation, respectively. For each method, the downstream networks are retrained separately using the same ground-truth annotations and task-specific losses. The learning rate, batch size, and training epochs are kept identical across methods to ensure a fair comparison. We have revised the manuscript accordingly to clarify these experimental settings.

The authors have adequately addressed my concerns. I am satisfied with their responses and will increase my rating accordingly.

AuthorsOpenReview ↗

Reply to Reviewer M4At

We greatly appreciate your kind recognition. Thanks again for your professional comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.