acceptodds
Withdrawn from ICLR 2027

Learning Task-Agnostic Representation for Multimodal Image Fusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent image fusion methods increasingly incorporate downstream tasks into joint optimisation, with the aim of injecting task-oriented semantic information into the fused image. However, compressing multimodal information into a single fused image inevitably imposes a fixed trade-off between the source modalities, making it difficult to accommodate the diverse information requirements of different scenes and downstream tasks. In this work, we explore an alternative paradigm. Rather than compressing complementary multimodal cues into a single fused representation, we aggregate source information under complementary fusion objectives to learn Task-Agnostic Representations (TAR). By allocating heterogeneous fusion cues that would otherwise compete within a shared output space, TAR preserves richer complementary information and provides a flexible intermediate representation for diverse downstream applications, while maintaining the same information budget as conventional image-fusion outputs. Importantly, TAR is learned without relying on task-specific annotations or downstream-task supervision, allowing the same representation to be directly exploited by different perception tasks. For conventional image fusion, a lightweight fusion head can further project TAR into a standard fused image, providing a flexible means of recovering visually faithful fusion results. Extensive experiments across multiple fusion and perception benchmarks demonstrate that TAR consistently improves downstream object detection and semantic segmentation performance while retaining competitive image-level fusion quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
infrared-visible fusion, multimodal representation learning, channel decoupling, task-agnostic learning, downstream perception, object detection, semantic segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KZk4OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper argues that the conventional single-output formulation of multimodal image fusion is fundamentally limited: compressing complementary infrared and visible information into one fused image forces intensity, gradient, and structure objectives to compete in a shared space, and the pseudo-RGB replication used to feed downstream networks only adds channels without clear functional roles. The proposed Task-Agnostic Representations (TAR) instead learn a K-channel output in which each channel is supervised by a dedicated objective (intensity, gradient, and structure), using a dense U-Net backbone, with no downstream supervision. A lightweight fusion head, trained with the backbone frozen, maps TAR back to a conventional fused image for image-level evaluation. The representation is architecture-agnostic: only the output head needs to be swapped to plug into CNN- and Transformer-based backbones, and downstream task performance improves in both cases. They achieve excellent performance on several image fusion benchmarks.

Critical strengths
  1. The central idea is simple yet effective: give each objective its own channel instead of packing everything into one fused image. This removes the forced trade-off between the two modalities and the redundancy of pseudo-RGB inputs, and it works in practice. The representation needs no downstream supervision and transfers across architectures; the authors only replace the output head of two unrelated backbones, one CNN and one Transformer, and both improve on detection and segmentation. Task-aware fusion methods, in contrast, are tied to a specific task and its labels. That this approach avoids such coupling makes the generality genuinely useful.

  2. The experiments are consistently in the paper's favor. TAR is first or second on every image-level metric over three fusion benchmarks, and it brings clear gains to downstream detection and segmentation, leading every object category on the segmentation benchmark. The ablations agree: the full representation beats single-channel replication and the best single-role variant, and the trend across channel configurations holds up. The improvement comes from explicit role assignment, not from adding channels.

  3. The design also keeps the method easy to use and to compare. The representation follows the standard three-channel input format, so off-the-shelf detectors and segmentation module can take it directly with no architectural changes; a lightweight head can map it back to a conventional fused image for image-level evaluation. The results therefore remain comparable with the large body of fusion literature, while the same representation serves downstream models.

Critical weaknesses
  1. The role of the fusion head and the two-stage protocol is left underexplored. Image-level metrics are all computed on the fusion-head output, yet there is no ablation isolating the head design, for example, sensitivity to λs or head capacity.

  2. The downstream comparison is somewhat one-sided: Table 3 compares TAR only against general-purpose fusion methods, while task-aware fusion approaches specifically designed for detection and segmentation are not included. Given that one of the paper's central claims is improving downstream perception without task-specific supervision, a direct comparison with such task-supervised alternatives would better clarify the positioning and relative advantage of the contribution.

  3. The ablation on the number of channels K is insufficiently informative. In the supplementary material, different K configurations are reported only as ranking tables, without the underlying downstream metric values, making it difficult to judge how large the performance differences between different channel settings actually are.

  4. The three-channel representation Z is what gets fed into the downstream networks, yet every image-level metric in the tables is computed on the fused image produced by the fusion head. This raises a question: does the head output actually tell us anything about the quality of Z itself? Since Z is the representation that detection and segmentation actually consume, why not evaluate image-level quality directly on Z?

  5. The paper positions itself against "task-aware fusion," but existing methods like UniFusion and GIFNet already describe themselves as unified or task-agnostic, and feeding fused outputs to downstream networks is hardly new. Given that, it would help to spell out in the related work what "task-agnostic" means here beyond these methods, for instance, those frameworks share one fusion paradigm across tasks, whereas TAR provides downstream networks with semantically role-specific input channels. It would also be worth clarifying whether the three channels are semantically meaningful to downstream models or mainly offer better input statistics. Making this distinction explicit would sharpen the contribution's boundaries.

Questions for authors
  1. Image-level metrics are all computed from the fusion-head output, while downstream models consume the three-channel representation Z directly. To what extent does the head output reflect the quality of Z, and was image-level quality considered when designing Z?

  2. Table 3 compares the proposed method only against general-purpose fusion methods. Which task-aware fusion baselines were considered, and how does the method compare against them on M3FD and FMB?

  3. The ablation on the number of channels K is reported as rankings without concrete values. Could you provide the actual mAP/mIoU for each K configuration to quantify the performance differences?

Reviewer_ZdW5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a three-channel representation, TAR, in which different channels are trained using intensity, gradient, and structural losses, respectively. The fusion backbone does not use downstream task labels, and its output can be used as input to standard detection and segmentation networks without changing their architectures, while a separately trained lightweight fusion head is responsible for generating the fused image for viewing. Experiments on fusion, detection, and segmentation, as well as results on different backbones, demonstrate that this representation improves performance on some downstream tasks while maintaining competitive fusion quality.

Critical strengths
  1. TAR processes representations intended for perception separately from those intended for viewing in the fused image. The fusion backbone does not require detection or segmentation labels, and its output is compatible with standard three-channel networks. Freezing the backbone before training the lightweight fusion head also prevents further alterations to the perceptual representations intended to improve image aesthetics. This design offers practical value.
  2. Table 1 shows that TAR improves both detection and segmentation on both tested fusion backbones. Taking UPTP as an example, the detection mAP50:95 improved from 0.564 to 0.585, and the segmentation mIoU improved from 0.430 to 0.490. The channel ablation results in Table 4 also support the role of preserving diverse information: the full TAR achieves a detection mAP50:95 of 0.584, which is higher than the 0.563 achieved with grayscale and chrominance inputs, as well as the 0.574 achieved with the best-performing single-role variant.
Critical weaknesses
  1. The existing ablation studies do not fully explain why it is necessary to assign intensity, gradient, and structural targets to three separate channels. While the full TAR outperforms grayscale and single-function inputs, it lacks comparisons with a matched three-output model in which each channel is supervised by all three losses, as well as with a simple source-based input such as [IR, VIS, mean(IR, VIS)]. Furthermore, assigning losses to different outputs does not imply that optimization interference within the shared backbone has been eliminated. Therefore, the current evidence is insufficient to fully support the benefits of this specific division and its interpretation.
  2. The reporting of segmentation results is incomplete. Tables 1 and 3 show only the six categories of FMB, without explaining the rationale for selecting these categories or providing the remaining categories. The scope of the mIoU calculation is also unclear: the average of the six per-class IoUs reported for TAR in Table 3 is approximately 0.467, while the reported mIoU is 0.452; this discrepancy may be related to the inclusion of other categories and requires clarification. Furthermore, the fusion metrics listed in Section 4.1 include MI, but Table 2 reports SSIM. The issues mentioned above limit the assessment of the overall benefits.
Questions for authors
  1. When the backbone, training budget, preprocessing, and downstream settings are kept consistent, how does TAR perform compared to the following inputs: all three output channels receiving all three losses, and the direct use of [IR, VIS, mean(IR, VIS)]? These controls would help distinguish between the benefits of the specific channel division and those of retaining multiple types of information.
  2. When the paper refers to “reducing optimization interference,” does this specifically refer to interference between output targets or interference in the shared backbone parameter updates? Aside from the lower loss shown in Figure 1, what other evidence supports this interpretation?
  3. Why are only six FMB classes reported? Please provide complete per-class IoU values and specify the class set used for mIoU, how background and ignored labels are handled, and the averaging rules. Do the improvements shown for certain classes come at the expense of declines in other classes?
  4. Which set of metrics—EN, MI, VIF, and SD in Section 4.1, or SSIM, VIF, EN, and SD in Table 2—were actually used for evaluation? Please provide a consistent description; if MI was indeed calculated, please include the corresponding results.
Reviewer_Bm1QOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Task-Agnostic Representations (TAR) for infrared–visible image fusion. Instead of forcing intensity, gradient, and structure objectives to compete within a single fused output, TAR learns a three-channel, role-specific representation: one channel for intensity/saliency, one for gradient/edges, and one for structure/layout. Each channel is supervised by a dedicated loss, reducing objective competition while preserving complementary multimodal cues. The representation keeps the standard three-channel input format, so it can be fed directly into off-the-shelf detection and segmentation networks without architectural changes. A lightweight fusion head can also project TAR back to a conventional single-channel fused image for visualization and image-level evaluation. Training is two-stage: first learn the representation, then freeze it and train only the fusion head.

Critical strengths

1.The task is unrelated, avoiding downstream annotation dependencies It indicates that the learning stage does not utilize detection/segmentation annotations, nor does it incorporate task-specific fusion losses. Therefore, it is more flexible than task-aware fusion methods, and the same representation can serve multiple downstream tasks.

2.Maintaining a standard three-channel interface ensures strong compatibility The TAR output remains in a three-channel format, allowing direct integration into existing networks such as YOLOv5 and SegFormer without requiring modifications to the downstream architecture. Additionally, it can be integrated into different fusion backbones by simply replacing the output head.

3.The architecture-independence is supported by experiments The paper reports improved detection and segmentation results by merely replacing the output heads on both CNN-based MMDRFuse and Transformer-based UPTP, indicating that the method is not tied to a specific fusion network.

4.Balancing machine perception and traditional fusion evaluation The lightweight fused head can project the three-channel representation into a traditional single-channel fused image for visualization and image-level metric evaluation. During the two-stage training, the backbone is frozen to avoid perturbation of downstream perceptual representations by the visualization target.

Critical weaknesses

1.The channel roles and losses are manually designed:The roles of intensity, gradient, and structure, as well as their loss functions, are manually specified and not learned; the paper also acknowledges that more principled or learnable channel decompositions are worth exploring. This limits the upper bound of representational power and the degree of automation.

2.K=3 is limited by the input from the downstream three channels:The method is naturally suitable for standard three-channel networks, but when K>3, the downstream input architecture needs to be modified, which has not been explored in the paper; this limits the advantage of "flexible channel count" in actual downstream deployment.

3.Direct visualization is unnatural:The three-channel encoding of heterogeneous information does not have a uniform grayscale appearance, and direct RGB visualization may appear unnatural; it must rely on an additional fusion head to obtain a visually reasonable fused image. The paper also explicitly mentions this point in the Limitations section.

4.Limited scene generalization verification:The experiments were primarily conducted in nighttime driving and surveillance scenarios, and only trained on LLVIP before being migrated to other benchmarks; there is insufficient verification of the generalization ability for broader imaging conditions, adverse weather, and different sensor registration errors.

5.Downstream tasks and network coverage are relatively narrow:Detection only uses YOLOv5, and segmentation only uses SegFormer; although two types of fused backbones have been verified, the types of downstream perception networks and task types are still limited, and it cannot be fully proven to be effective for any perception model.

6.Traditional integration quality is not comprehensively leading:The method presented in Table 2 is competitive in some indicators, but not the best in all; the paper emphasizes "competitiveness" rather than overall optimality. The fusion head adopts a combination of maximum intensity prior and SSIM, which is relatively simple and may limit the upper bound of traditional fusion maps.

7.The three-channel semantics may still contain redundancy and lack quantitative analysis:The paper employs ablation studies to demonstrate the effectiveness of role division, but it fails to systematically quantify the correlation, redundancy, or information complementarity among the three channels; there is still a lack of in-depth analysis on whether the channels are strictly decoupled.

8.Strong assumption on shared structure:The structural channel employs SSIM to simultaneously constrain infrared and visible light, implicitly assuming that the two modalities share global layout and geometric consistency; this assumption may not hold when there is significant registration error or pronounced scene differences.

9.The training details and ablation scale are limited:The main training is set to only 10 epochs per stage by default, while ablation is conducted for 200 epochs on M³FD/YOLOv5. The ablation of K is mainly presented in graphical form, lacking more granular numerical values and statistical significance analysis.

Questions for authors

1.In Table 2, does “best overall performance” hold under matched training budgets, parameter counts, and loss-weight tuning for the fusion head versus all baselines?  2.Are the three TAR channels quantitatively shown to be non-redundant—e.g., via correlation, mutual information, or channel-ablation complementarity—beyond the downstream mAP ablation?  3.In the downstream M³FD/FMB experiments, is the TAR representation frozen during detector/segmentation training, or is it fine-tuned? If fine-tuned, how is “task-agnostic” distinguished from ordinary end-to-end adaptation?  4.In Table 1, are the +Ours and backbone baselines trained with identical downstream budgets, seeds, augmentation, and hyperparameter search, especially given the large UPTP segmentation gain?  5.In the K-ablation, are K=1/2/3 variants matched for total loss weight, output-head capacity, and channel-replication protocol, so that the gain can be attributed to role specialization rather than capacity or loss scaling?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.