Difference is the Signal: Modality Residual Curves for Frequency-Adaptive Infrared and Visible Fusion

The fusion of infrared and visible images is typically viewed as the integration of complementary information from the two modalities. However, when the two modalities provide inconsistent responses to the same scene, fusion models face another challenge: how to explicitly incorporate this cross-modal discrepancy into the fusion decision without ground-truth. Cross-modal residual provides direct but incomplete evidence. Residual energy can describe how strong the discrepancy is and which frequency regions it is concentrated, but it cannot provide directional information.To address this limitation, we propose the Modality Residual Curve (MRC), which combines two frequency variables with distinct roles: the discrepancy-magnitude curve, which characterizes the strength of cross-modal divergence across each frequency band; and the modality-direction curve, which compares the structural responses of the two modalities within the same frequency band. Their product forms a signed frequency prior. Its magnitude represents the strength of the intervention routing in that frequency band, while its sign indicates the direction of relative structural support rather than prescribing which modality the final fusion result must choose. The projected signed prior is added as an explicit bias to the content-driven expert routing scores. A dual-branch architecture aggregates the resulting modality-guided responses. Experiments on four public datasets, downstream applications, component ablations, and controlled degradation analyses consistently support the effectiveness and interpretability of the proposed method.