acceptodds
Withdrawn from ICLR 2027

Difference is the Signal: Modality Residual Curves for Frequency-Adaptive Infrared and Visible Fusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The fusion of infrared and visible images is typically viewed as the integration of complementary information from the two modalities. However, when the two modalities provide inconsistent responses to the same scene, fusion models face another challenge: how to explicitly incorporate this cross-modal discrepancy into the fusion decision without ground-truth. Cross-modal residual provides direct but incomplete evidence. Residual energy can describe how strong the discrepancy is and which frequency regions it is concentrated, but it cannot provide directional information.To address this limitation, we propose the Modality Residual Curve (MRC), which combines two frequency variables with distinct roles: the discrepancy-magnitude curve, which characterizes the strength of cross-modal divergence across each frequency band; and the modality-direction curve, which compares the structural responses of the two modalities within the same frequency band. Their product forms a signed frequency prior. Its magnitude represents the strength of the intervention routing in that frequency band, while its sign indicates the direction of relative structural support rather than prescribing which modality the final fusion result must choose. The projected signed prior is added as an explicit bias to the content-driven expert routing scores. A dual-branch architecture aggregates the resulting modality-guided responses. Experiments on four public datasets, downstream applications, component ablations, and controlled degradation analyses consistently support the effectiveness and interpretability of the proposed method.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Modality Residual, Frequency-Adaptive Fusion, Infrared and Visible Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bFohOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In view of the core view that the difference between infrared and visible images can be used as signals in the research of visible infrared image fusion, this paper proposes how to better use the frequency domain information to effectively model the contribution of dual-mode images, and realize the effective autonomous fusion of visible infrared images. Based on the two frequency-domain curves proposed by the author, one represents the difference intensity and the other represents the fusion difference preference, the network can dynamically determine whether to believe in infrared or visible images in different frequency bands, and realize the final fusion. The paper evaluates the performance on four public fusion datasets and two downstream tasks, and obtains excellent results.

Critical strengths

1. aiming at the topic of the selection of fusion strategy in the field of visible and infrared image fusion, the paper chooses to fully mine the information of modal differences. The advantages of infrared and visible light will change with the scene, frequency and degradation degree, so it is of practical significance to adjust the fusion strategy according to the frequency band. It is more complete to separate "how much difference" and "which modal structure has stronger response" than to use only residual strength.

2. the Modality Residual Curve (MRC) proposed in this paper is easy to understand and is a delicate design, which improves the interpretability of the fusion strategy in image fusion, which is enlightening for other researchers.

3. the paper gives the results of four public data sets, which makes the results of the paper more convincing.

4. in addition to the necessary comparative experiments, computational efficiency experiments, ablation experiments, and downstream task experiments, the paper also launched the controlled behavioral analysis, which is a commendable design.

Critical weaknesses

1. in Figure 6 (b), page 9, plots rB=M(b)D(b)r_B=M(b)D(b) as the visible input is blurred. Section 4.5 describes this as a decrease in "reliance on the visible modality." The figure also shows qualitative fused outputs, but it does not measure routing weights, cross-attention contributions, or causal output sensitivity to either input. Its curve demonstrates that the designed prior responds to a particular blur example; it does not establish the claimed change in the learned network's modality reliance or aggregate robustness.

2. the pipeline structure given in the paper is very simple, but the corresponding code is not provided.

3. in Figure 3 of the paper, it can be clearly seen that for an area with a large proportion of sky such as FMB dataset, the method of the paper will make the sky dark, which proves that the method proposed in the paper can not completely achieve the modal selection in line with human observation.

4. the relevant discussion in the paper is relatively thin in the selection of loss function, and there is no additional control of the loss of the influence of MRC curve on fusion, which weakens the persuasiveness of the model and may even lead to overclaim.

Questions for authors

1. Tables 1 and 2 contain 218 baseline values identical to those in TDFusion (CVPR 2025), with only six changed entries. Please identify which scores were imported and which were independently recomputed, explain the six differences, and clarify the statement that SegFormer and YOLOv8 were retrained. For recomputed results, please provide the evaluation configuration, checkpoint or prediction provenance, run logs, and seeds.

2. Please explain the first point in Critical Weaknesses.

3. Can the author provide a code demo to better understand the proposed MRC?

4. Can the author give a more detailed discussion on points 3 and 4 in Critical Weaknesses?

5. The author uses MoE to assist in the generation of fused images. Do you use balance loss in the process of using MoE?

6. The author's pipeline structure is very simple, but the parameter quantity is not small. Are there some parts omitted in Figure 2?

Reviewer_dNVfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Modality Residual Curve (MRC), a frequency-adaptive framework for infrared–visible image fusion. Motivated by the observation that cross-modal residual energy captures discrepancy magnitude but lacks directional information, the method introduces two frequency-dependent curves: a discrepancy-magnitude curve measuring normalized residual energy and a modality-direction curve estimating relative structural responses using gradient information. Their product forms a signed frequency prior, which guides adaptive frequency-band partitioning and explicitly biases mixture-of-experts routing. The framework employs two parameter-unshared, role-swapped branches, whose outputs are integrated through cross-attention. The proposed method is evaluated on four public datasets (MSRS, FMB, M3FD, and LLVIP) using six fusion quality metrics. Additional experiments include semantic segmentation, object detection, component ablations, computational efficiency comparisons, and controlled blur analyses. The reported results demonstrate competitive fusion performance and improvements on several downstream evaluation metrics.

Critical strengths
  1. Clearly motivated problem formulation. The paper identifies a meaningful limitation of using cross-modal residual energy alone for fusion decisions, namely its inability to distinguish the direction of relative structural responses. Separating discrepancy magnitude from modality direction provides a clear motivation for the proposed framework.
  2. Useful component ablations and controlled behavioral analysis. The paper examines the contributions of the magnitude and direction curves, adaptive frequency partitioning, and dual-branch design. The controlled blur analysis further illustrates how the constructed prior responds to changes in visible-image quality, providing useful observations about the behavior of the proposed mechanism.
Critical weaknesses
  1. The training objective is fundamentally underdetermined. The spatial and frequency-domain L1 losses in Eq. (4) admit an entire family of globally optimal solutions consisting of convex combinations of the two inputs, including direct copies of either modality. Consequently, the objective does not distinguish meaningful fusion decisions from trivial modality reproduction. The paper does not explain how its optimization formulation supports the claimed discrepancy-aware attribution mechanism.
  2. The central modality-attribution claim is insufficiently established. The proposed signed prior directly modifies frequency-expert routing logits rather than modality-specific contribution weights. Although this provides an explicit routing bias, neither the theoretical formulation nor the experiments establish that the sign of the prior produces the intended change in actual modality contribution. The controlled analysis in Fig. 6 examines prior responses rather than verifying the claimed fusion decisions.
  3. Limited justification of structural preference and insufficient differentiation from prior work. The modality-direction curve relies on relative gradient energy, which is sensitive to contrast scaling and noise and does not necessarily indicate more reliable structural evidence. Furthermore, the proposed architecture shares several important design elements with existing residual-prior and frequency-domain fusion frameworks, particularly RPFNet, without a direct comparison demonstrating the benefits of its distinctive signed-routing mechanism.
Questions for authors
  1. Regarding the training objective in Eq. (4): The spatial and frequency-domain L1 losses appear to admit the same global minimum for any convex combination of the two input images, including direct reproduction of either modality. Could the authors clarify what prevents such trivial solutions and how the proposed objective encourages meaningful discrepancy-aware fusion decisions?

  2. Regarding the signed prior and modality attribution: Eq. (16) applies the signed prior to frequency-expert routing logits rather than directly to modality-specific weights. Could the authors explain or demonstrate how the sign of the prior translates into the intended preference for the source or reference modality in the final fused output? In particular, does the actual modality contribution change consistently with the prior under the controlled degradation setting in Fig. 6?

  3. Regarding the reliability of the modality-direction curve: Eq. (9) uses relative gradient energy to estimate structural preference. Since gradient energy can increase under noise or contrast changes without improving structural reliability, what evidence supports its use as a reliable modality-direction indicator? How does the proposed prior behave when one modality contains strong noise-induced gradients?

  4. Regarding novelty and comparison with closely related work: The proposed framework shares residual-prior modeling, frequency-domain processing, and dual-branch design elements with existing methods, particularly RPFNet. Could the authors clarify the essential methodological distinction and provide evidence that the signed prior-guided routing mechanism offers improvements beyond these existing designs under comparable experimental settings?

Reviewer_ERUwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Modality Residual Curve (MRC)-based method for infrared-visible image fusion. The proposed method introduces a discrepancy-magnitude curve and a modality-direction curve to construct frequency priors, which are subsequently used to guide the weighting of multiple frequency experts. The effectiveness of the proposed method is preliminarily evaluated on four public datasets.

Critical strengths
  1. Compared with approaches that rely entirely on implicit features for expert selection, explicitly incorporating the discrepancy-magnitude curve and the modality-direction curve into the expert routing process provides a certain degree of interpretability.
  2. The manuscript provides a partial analysis of the controlled behavior, examining band-wise modality preferences within a selected region and the response of MRC to progressively introduced blur.
Critical weaknesses

1.The manuscript lacks sufficient justification for the necessity of introducing the discrepancy-magnitude curve and the modality-direction curve. 2. The effectiveness of the constructed explicit signed prior in guiding expert routing decisions has not been adequately validated. 3. The manuscript lacks an ablation study without the explicit signed prior under identical training conditions. 4.The manuscript lacks comparisons with recent representative image fusion methods proposed in 2026.

Questions for authors
  1. Why were only the discrepancy-magnitude curve and the modality-direction curve selected? Have the authors evaluated the effectiveness of other similar curves?
  2. How does the explicit signed prior influence expert routing decisions, and does this guidance consistently improve fusion performance?
  3. How does the model perform without the explicit signed prior under identical training conditions?
  4. It is recommended that the authors provide additional comparisons with recent representative image fusion methods proposed in 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.