acceptodds
Under review as a conference paper at ICLR 2027

PTFusion: Photometric and Texture Prior Learning for Illumination-Adaptive Infrared and Visible Image Fusion

Abstract

The fusion of infrared and visible images in low-light environments faces the problem of degraded photometric and texture information. The existing methods usually couple photometric restoration with texture compensation, which limits their adaptability to different lighting conditions. To address these issues, we propose PTFusion, a multi-illumination image fusion method based on photometric and texture priors. Specifically, we use images under multiple lighting conditions to learn photometric priors, mapping different lighting states to a unified photometric representation space to enhance the model's adaptability to different lighting conditions. At the same time, we construct a texture prior network to inject texture features into the fusion feature space to compensate for the lack of texture information and insufficient local structural details caused by lighting degradation. In the fusion process, we use photometric priors to adapt visible features and inject texture priors to compensate for lighting degradation and texture deficiencies. Then, prior of photometric and texture is used to supervise the fusion results, improving the photometric fidelity and texture details of the fused image. The experimental results show that our method has excellent multi-light adaptation ability and exhibits competitive comprehensive performance in fusion tasks and downstream visual tasks. The code will be released publicly.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image fusion, Infrared and visible image, Low-light image

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xaYjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the issues with existing infrared and visible light image fusion methods—namely, that most are only suitable for low-light environments, and that luminance restoration and texture compensation are mutually coupled, leading to unstable performance under variable lighting conditions—by proposing PTFusion. This method decouples the luminance prior from the texture prior; the luminance prior maps images taken under different lighting conditions to a unified, normal-illumination feature space to achieve luminance adaptation, while the texture prior uses a multi-gradient operator to extract high-frequency details for texture compensation and incorporates object-aware constraints to preserve infrared object information. The fusion process utilizes both priors to simultaneously perform feature injection and loss supervision.

Critical strengths

The paper proposes a model for image fusion under low-light, normal-light, and high-light conditions. The paper is well-written and flows smoothly overall.

Critical weaknesses

There are significant issues with ablation. The article does not adequately explain the theory.

Questions for authors
  1. Equation 8 requires a more detailed explanation. Why does the network need to extract texture information? Wouldn’t it be better and more accurate to obtain it directly using the Sharpe, Sobel, LOG, and Laplacian operators?
  2. Your ablation results are shown in Figure 4, not Figure 3. There is an error here. The authors need to provide results based on objective metrics to demonstrate the validity of the ablation. Wouldn’t it be better to directly inject texture information into the features rather than using texture information extracted by the network? The authors need to include comparative experiments in the ablation analysis. Additionally, experimental results under high-exposure conditions should be provided.
  3. In the texture information injection module, the model selected four gradient operators. However, in the loss function, the model only calculates the loss with the Sobel operator, which is unreasonable.
3 comments

R1

Traditional operators are used to extract texture responses that already exist in images, while our texture prior network aims to learn texture compensation representations under lighting degradation conditions. When an image is affected by degradation such as low light or overexposure, some texture information may be weakened, and it is difficult to effectively compensate for these degraded information by directly applying fixed operators. Therefore, we introduce a learnable texture prior network to learn texture compensation representations under different lighting conditions.

Formula (8) can be further expanded as:

LT=LS+LC+LL+LG\mathcal{L}_T = \mathcal{L}_S + \mathcal{L}_C + \mathcal{L}_L + \mathcal{L}_G

where the four terms are defined as:

  • Sobel: LS=C(G^S−Sobel(IN))\mathcal{L}_S = \mathcal{C}(\hat{G}_S - \mathrm{Sobel}(I_N))
  • Scharr: LC=C(G^C−Scharr(IN))\mathcal{L}_C = \mathcal{C}(\hat{G}_C - \mathrm{Scharr}(I_N))
  • Laplacian: LL=C(G^L−Laplacian(IN))\mathcal{L}_L = \mathcal{C}(\hat{G}_L - \mathrm{Laplacian}(I_N))
  • LoG: LG=C(G^G−LoG(IN))\mathcal{L}_G = \mathcal{C}(\hat{G}_G - \mathrm{LoG}(I_N))

Where:

  • INI_N: The corresponding normal-light reference image.
  • G^S,G^C,G^L,G^G\hat{G}_S, \hat{G}_C, \hat{G}_L, \hat{G}_G: The texture responses predicted by the four branches.
  • Sobel, Scharr, Laplacian, and LoG: Four fixed texture operators.
  • C\mathcal{C}: The Charbonnier loss function.

R2

Thank you to the reviewer for your careful review and valuable suggestions. We will correct this issue in the revised draft and check the relevant captions.

Due to the limitation of the original manuscript length, we mainly presented the qualitative ablation results of photometric and texture priors through Figure 4. We have added quantitative ablation experiments.

LowlightW/O PTW/O PW/O TOurs
PSNR61.53461.73762.09361.233
CC0.60580.60280.56830.5489
Nabf ↓0.05010.09250.12340.1919
SSIM0.79090.80820.87360.8855
MS-SSIM0.74320.75890.87270.8945
FMI0.88930.87850.88140.8781
OverexposureW/O PTW/O PW/O TOurs
PSNR60.88658.91661.22360.029
CC0.53670.50830.53000.5121
Nabf ↓0.11660.19290.15070.2128
SSIM0.89130.90350.89830.9066
MS-SSIM0.88880.91650.90130.9197
FMI0.86900.86700.87370.8679
MetricsW/O PTW/O PW/O TOurs
ΔPSNR ↓0.64792.82080.87031.2040
ΔCC ↓0.06920.09450.03820.0368
ΔNabf ↓0.06660.10040.02730.0209
ΔSSIM ↓0.10040.09540.02470.0211
ΔMS-SSIM ↓0.14560.15760.02870.0252
ΔFMI ↓0.02030.01150.00770.0102

Partial ablation configurations achieved higher results in evaluation metrics, which is related to PTFusion's photometric recovery and texture compensation mechanisms. These operations will alter the brightness, contrast, and local texture response of the fused image relative to the original source image, thereby affecting some evaluation metrics based on source image consistency. Therefore, changes in a single indicator cannot fully reflect the impact of photometric restoration and texture compensation on fusion quality. The goal of PTFusion is to improve the photometric adaptability and structural preservation ability of fusion results under complex lighting conditions through the synergistic effect of photometric and texture priors.

Under complex lighting conditions, our method exhibits smaller fluctuations in cross illumination performance, demonstrating the good stability of the dual prior mechanism in maintaining structure.

The visual results in Figure 4 demonstrate the role of photometric priors in improving the visibility of degraded areas, as well as the contribution of texture priors in compensating for local details.

R3

Thank you very much to the reviewer for providing this valuable suggestion. Based on the suggestions of the reviewers, we further improved the texture supervision mechanism in the fusion stage by extending the original Sobel single operator gradient constraint to a multi operator joint supervision of Sobel, Scharr, Laplacian, and LoG, allowing different types of edge and texture information to jointly participate in the optimization of the fusion network. We have conducted supplementary experiments on this improvement, and the experimental results show that the introduction of multi operator joint supervision further improves the fusion performance of the model, fully reflecting the positive role of this suggestion in improving our method. We will supplement the improved experimental results in the revised manuscript and express our sincere gratitude to the reviewers for their constructive suggestions in the official version.

The author answered my question.

Thank you very much for the professional, detailed, and constructive comments provided by the reviewer during the review process.

We particularly appreciate your valuable suggestions regarding the texture supervision mechanism. Your comments are highly insightful and have provided important guidance for us to further improve and strengthen our work. Based on your suggestions, we have added the corresponding theoretical explanations and experimental results to the revised manuscript and further improved the relevant content. We sincerely appreciate your constructive suggestions and the valuable guidance they have provided for our work.

Thank you again for your careful and professional review. Your valuable feedback has been very helpful in further improving the methodology, technical presentation, and experimental validation of our paper. Therefore, we sincerely hope that you could reconsider your overall evaluation and score of the paper based on the revised manuscript and the additional experimental results. If you find that our revisions and additional results have adequately addressed your concerns, we would be very grateful if you could consider raising the score accordingly.

Thank you again for your careful review and valuable constructive feedback.

Reviewer_ti5YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PTFusion, an illumination-adaptive infrared–visible image fusion method based on photometric and texture priors. A two-stage photometric prior network learns representations across different lighting conditions, while a multi-operator texture prior network compensates for missing structural details. Both priors guide feature fusion and supervision, with target-aware constraints preserving infrared information. Experiments on multiple benchmarks and downstream segmentation tasks demonstrate competitive fusion performance under diverse illumination conditions.

Critical strengths
  1. Well-motivated dual-prior framework. PTFusion separately addresses photometric variation and texture degradation. The two-stage photometric learning and multi-operator texture extraction provide complementary guidance, while target-aware constraints help preserve infrared structures.

  2. Comprehensive experimental evaluation. The experiments cover MSRS, M3FD, LLVIP, and DDL, with comparisons against nine fusion methods. Table 1 demonstrates competitive performance across different illumination conditions. Table 2 further reports the highest segmentation mIoU of 64.03, supporting the effectiveness of the proposed framework for downstream perception.

Critical weaknesses
  1. Limited quantitative ablation analysis. Figure 4 qualitatively illustrates the effects of removing photometric and texture priors, but their individual contributions are not quantitatively evaluated. The effects of different texture operators and target-aware constraints also remain unclear.

  2. Insufficient methodological details. The construction of infrared target masks and the preparation of multi-illumination supervision could be clarified. Additionally, the definition of the SSIM loss in Equation (15) is somewhat ambiguous, affecting the clarity of the training objective.

Questions for authors
  1. Figure 4 provides qualitative ablations for the two priors, but their contributions are not quantitatively evaluated. A small ablation table reporting the performance of each prior separately would make the results more convincing.

  2. The photometric and texture priors are pretrained on MEF, while the fusion network is trained on MSRS. How are the different illumination inputs and their corresponding supervision constructed during pretraining?

  3. The target-aware constraints are intended to protect infrared structures during texture compensation. How are the target masks generated, and are they required during inference?

  4. Table 1 shows strong overall results, but DAFusion achieves higher SSIM and MS-SSIM on LLVIP. It would be interesting to understand why PTFusion performs less favorably on this dataset despite its illumination-adaptive design.

  5. Equation (15) defines the structural similarity loss as a sum of SSIM terms, which is then included in the minimized objective in Equation (18). Is the actual implementation based on 1 − SSIM, or is a different formulation used?

1 comment

R1

Thank you for the valuable suggestions from the reviewer. Due to the limitation of the original manuscript length, we mainly presented the qualitative ablation results of photometric and texture priors through Figure 4. We further provided quantitative ablation results.

LowlightW/O PTW/O PW/O TOurs
ΔPSNR ↓0.64792.82080.87031.2040
ΔCC ↓0.06920.09450.03820.0368
ΔNabf ↓0.06660.10040.02730.0209
ΔSSIM ↓0.10040.09540.02470.0211
ΔMS-SSIM ↓0.14560.15760.02870.0252
ΔFMI ↓0.02030.01150.00770.0102

The goal of PTFusion is to improve the photometric adaptability and structural preservation ability of fusion results under complex lighting conditions through the synergistic effect of photometric and texture priors. Under complex lighting conditions, our method exhibits smaller fluctuations in cross illumination performance, demonstrating the good stability of the dual prior mechanism in maintaining structure. We will further analyze the roles of the two priors in the revised manuscript.

R2

We use different exposure images of the same scene in the MEF dataset to construct multi light training samples, including low light, normal light, and high light images.

For photometric priors, we adopt a two-stage training strategy. In the first stage, each lighting input takes its own image as the supervised target, while optimizing the encoder and decoder to learn the image reconstruction ability under different lighting conditions. In the second stage, each lighting input is supervised with a normal lighting image as the target, and only the encoder is updated to learn the image reconstruction ability under different lighting conditions.

Stage 1: (Eq. 4)

I^d=DP(EP1(Id)),d∈{low,normal,high}.\hat{I}_d = D_P\left(E_P^{1}(I_d)\right), \quad d \in \left\lbrace \mathrm{low}, \mathrm{normal}, \mathrm{high} \right\rbrace.

Stage 2: (Eq. 5)

I^normal(d)=DPfrozen(EP2(Id)),d∈{low,normal,high}.\hat{I}_{\mathrm{normal}}^{(d)} = D_P^{\mathrm{frozen}}\left(E_P^{2}(I_d)\right), \quad d \in \left\lbrace \mathrm{low}, \mathrm{normal}, \mathrm{high} \right\rbrace.

For texture priors, we use the same multi light image as input and use the corresponding normal light image as the supervised target to learn the prior texture features corresponding to different lighting conditions. . After completing pre training, we fixed the parameters of two types of prior networks and trained the fusion network on the MSRS dataset. MEF is mainly used to learn photometric and texture representations across lighting conditions, while MSRS is used to train the fusion process of infrared and visible image, thereby achieving fusion of infrared and visible images under different lighting conditions.

R3

We use object detection networks to obtain target regions in images and construct corresponding target masks based on the detection results. This mask is used for calculating the target perception loss during the fusion network training phase. The target mask is only used for loss calculation during the training phase and does not directly participate in the forward feature injection of the fusion network. There is no need to input additional target masks or run additional detection networks during the inference process. We will supplement the specific configuration of the detection model, the target mask generation method, and its role in the loss function in the revised manuscript.

R4

PTFusion aims to improve the fusion adaptability of the model under different lighting conditions such as low light, normal light, and high exposure through the synergistic effect of photometric and texture priors.

We learn cross illumination representation through photometric priors and compensate for the local structure of degraded regions using texture priors. For low light nighttime images in LLVIP, this photometric adaptation and texture compensation will alter the local brightness, contrast, and texture response of the fusion result relative to the original source image, thereby affecting the indicators based on source image consistency. Combining visual results for comprehensive analysis, our fused image has better visual effects.

Therefore, the differences in some indicators on LLVIP do not negate the cross illumination fusion ability of PTFusion.

R5

The SSIM in formula (15) represents structural similarity loss, and the actual calculation uses 1-SSIM. We will clearly explain its specific calculation method in the revised draft.

Reviewer_VdtUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PTFusion, an illumination-adaptive infrared and visible image fusion framework based on photometric and texture priors. The proposed method learns photometric representations across different illumination conditions and extracts texture priors to compensate for degraded texture information. Extensive experiments on multiple datasets demonstrate the effectiveness of the proposed method.

Critical strengths
  • The paper is well-organized.
  • The experimental results demonstrate competitive performance on multiple benchmarks.
Critical weaknesses
  • Insufficient ablation studies.
  • Insufficient explanation of some key technical details.
Questions for authors
  • The annotations in Fig. 1 appear to be unclear, making it difficult to intuitively follow the framework described in Section 3.1.

  • The ablation studies are insufficient. Section 4.3 mainly presents visual comparisons after removing the photometric and texture priors. More quantitative ablation studies are needed to evaluate the individual contributions of the proposed modules.

  • The implementation of the target perception constraints is not sufficiently explained. How are the target masks obtained, and do they require additional annotations or pretrained models? The authors should further clarify how these masks are incorporated into the fusion process.

  • More technical details should be provided to improve the clarity of the proposed method. For example: (1) What are the architectures of the encoders and decoders used in the proposed framework? (2) In Eq. (5), does DPD_P represent a decoder? Although Eqs. (4) and (5) use the same inputs and architecture, their training objectives and outputs differ. The authors should further clarify the differences in the training objectives and input-output relationships between the two stages. (3) In Eqs. (4) and (5), it is unclear whether images under different illumination conditions are processed independently or jointly. (4) The necessity of freezing the decoder in the second training stage should be further investigated. (5) The relationship between Eqs. (7) and (8) requires further clarification. Is Eq. (8) the loss function used to train the texture encoder?

  • Since the proposed method involves multiple training stages, does it introduce significantly higher training overhead compared with existing methods? A more detailed analysis of the training costs would be helpful.

1 comment

We thank the reviewer for the constructive comments and suggestions.

R1

We will reorganize Figure 1 in the revised manuscript, clearly labeling the input and output of each stage, the connection relationship between the encoder and decoder, trainable and frozen modules, as well as the process of prior feature injection and loss supervision, and synchronously improving the description in Section 3.1.

R2

Due to the limitation of the original manuscript length, we mainly demonstrated the visual effect after removing the luminance prior and texture prior through Figure 4. We further provide quantitative ablation results.

LowlightW/O PTW/O PW/O TOurs
ΔPSNR ↓0.64792.82080.87031.2040
ΔCC ↓0.06920.09450.03820.0368
ΔNabf ↓0.06660.10040.02730.0209
ΔSSIM ↓0.10040.09540.02470.0211
ΔMS-SSIM ↓0.14560.15760.02870.0252
ΔFMI ↓0.02030.01150.00770.0102

The goal of PTFusion is to improve the photometric adaptability and structural preservation ability of fusion results under complex lighting conditions through the synergistic effect of photometric and texture priors. Under complex lighting conditions, our method exhibits smaller fluctuations in cross illumination performance, demonstrating the good stability of the dual prior mechanism in maintaining structure. We will further analyze the roles of the two priors in the revised manuscript.

R3

We use object detection networks to obtain target regions in images and construct corresponding target masks based on the detection results. This mask is used for calculating the target perception loss during the fusion network training phase. The target mask is only used for loss calculation during the training phase and does not directly participate in the forward feature injection of the fusion network. There is no need to input additional target masks or run additional detection networks during the inference process. We will supplement the specific configuration of the detection model, the target mask generation method, and its role in the loss function in the revised manuscript.

R4(1)

Thank you to the reviewer for pointing out the issue of insufficient detailed description of the network structure. The original manuscript mainly focuses on prior learning and fusion mechanisms, thus simplifying the description of some network structures. We will supplement relevant information in the revised draft.

Q4(2)

The prior learning of luminosity adopts a two-stage training strategy, and the network structure of both stages is consistent. In the first stage, the encoder and decoder parameters are jointly updated to enable the network to learn image reconstruction capabilities under different lighting conditions. In the second stage, the decoder parameters are fixed, and only the encoder parameters are updated. Images under different lighting conditions are restored to their corresponding normal light targets to learn the photometric priors.

Stage 1 (Eq. 4)

I^d=DP(EP1(Id)),d∈{low,normal,high}.\hat{I}_d = D_P\left(E_P^{1}(I_d)\right), \quad d \in \left\lbrace \mathrm{low}, \mathrm{normal}, \mathrm{high} \right\rbrace.

Stage 2 (Eq. 5)

I^normal(d)=DPfrozen(EP2(Id)),d∈{low,normal,high}.\hat{I}_{\mathrm{normal}}^{(d)} = D_P^{\mathrm{frozen}}\left(E_P^{2}(I_d)\right), \quad d \in \left\lbrace \mathrm{low}, \mathrm{normal}, \mathrm{high} \right\rbrace.

Q4(3)

Phase 1: Low light input reconstruction of low light, normal light input reconstruction of normal light, and high light input reconstruction of high light.

Phase 2: Low light input to reconstruct normal light, normal light input to reconstruct normal light, high light input to reconstruct normal light.

Q4(4)

We freeze the decoder in the second stage mainly to focus the cross illumination adaptation process on the photometric encoder. If the encoder and decoder are updated simultaneously in the second stage, it is difficult to clearly constrain cross illumination adaptation, which mainly occurs in the encoder. The purpose of adopting this training strategy is to learn a photometric prior encoder that can be transferred to subsequent fusion tasks.

Q4(5)

Formula (7) describes the process of extracting prior texture features using a texture prior network. Formula (8) is a multi operator texture supervised loss used for training texture prior networks. We will further clarify the formula content in the revised draft.

Q5

The prior knowledge only needs to be pre trained once, and the trained prior network can be uniformly applied to fusion tasks under different lighting conditions, without the need to retrain corresponding prior models for each lighting environment. Overall, PTFusion adopts a method of separating one-time prior learning from subsequent fusion training, exchanging additional offline pre training costs for feature representations with cross illumination adaptability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.