acceptodds
Under review as a conference paper at ICLR 2027

From Unpaired Priors to Paired Calibration: T2IR for RGB-to-Infrared Image Translation

Abstract

RGB-to-infrared (IR) image translation is difficult because visible reflected light and thermal radiation follow fundamentally different physical processes. Paired RGB/IR data are also scarce and frequently only weakly aligned, so a single-paradigm approach often fails to jointly maintain structural correspondence and IR appearance diversity. To tackle these issues, we propose T2IR, a two-stage diffusion framework that moves from unpaired prior learning to paired structural calibration. The first stage exploits separate RGB and IR image pools to acquire a cross-modal infrared appearance prior. The second stage uses paired supervision to adjust geometric structure and infrared intensity. We additionally apply training-time regularization on thermal sources and texture statistics, which mitigates underheated thermal regions and visible-texture leakage from RGB images. To support this framework, we construct THIRAlign, a synchronized paired RGB/IR dataset obtained with a beam-splitter-assisted dual-modal acquisition system and calibration-based spatial alignment. Experiments on three datasets demonstrate that T2IR surpasses six state-of-the-art methods. The results verify that unpaired infrared prior learning and paired structural calibration are complementary, and that the proposed training-time regularization compensates for the weakness of global image reconstruction losses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RGB-to-infrared translation, image-to-image translation, diffusion models, cross-modal image translation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4TgfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a two-stage framework for RGB-to-IR image translation. The first stage adopts a CycleGAN-style bidirectional unpaired translation scheme to learn cross-modal appearance priors, while the second stage retains the RGB-to-IR path and fine-tunes it using paired data. The paper also introduces the THIRAlign paired RGB–IR dataset and several training losses targeting thermal-source responses and RGB-texture leakage. Overall, the dataset appears to be an important contribution of the paper. On the method side, the framework mainly builds on an existing pretraining and fine-tuning paradigm, with its main additions being heuristic loss designs. There remains substantial room to strengthen the experimental validation, dataset documentation, and method presentation.

Critical strengths

Please refer to the Paper Summary Section

Critical weaknesses

1.The compact-source loss is based on a simple heuristic: it selects small bright regions in the target IR image using fixed rules and penalizes the model when these regions are predicted too dark. In the ablation table, the compact-source loss is removed together with another bright-region loss. Therefore, it is unclear how much improvement comes from the compact-source loss itself. As an additional comparison, the authors may consider using a conventional visual saliency map or local-contrast map to define the important thermal regions. This could help validate the advantage of the proposed compact-source mask.

2.THIRAlign appears to be one of the paper’s main contributions, yet its collection and alignment procedure are only briefly described. The appendix should provide detailed acquisition and calibration/alignment procedures, quantitative alignment errors, and comparisons to existing paired RGB–IR datasets. Figure 1 is not informative in its current form: any paired RGB–IR dataset can show corresponding RGB/IR examples. The paper should also report the number of independent scenes, sequences, capture durations and environmental conditions. 32k image pairs may contain substantial temporal redundancy if they originate from a small number of slowly changing scenes.

3.The mask is defined by regions with a strong RGB Sobel response but an IR response below a fixed quantile; this is not equivalent to the texture exists in RGB but absent in IR. It can also include valid but moderately strong thermal edges. The authors should provide sensitivity analyses for the Sobel quantile, loss weight, and mask coverage, together with qualitative examples of the selected regions.

4.In Figure 3, especially the M3FD and KAIST examples, the advantage over strong baselines is not visually clear. Moreover, the appendix provides additional quantitative ablations but no corresponding qualitative ablations.

Questions for authors

5.The first stage largely adopts a conventional CycleGAN-style bidirectional unpaired translation objective on a latent diffusion backbone, while the second-stage paired RGB-to-IR calibration appears to be the main technical component. The paper should better distinguish an effective pretraining recipe from a methodological contribution, reduce the emphasis on Stage 1, and compare against stronger alternative ways to exploit unpaired IR data.

6.As an additional comparison, it would be useful to evaluate direct Sobel/gradient supervision between generated and target IR images. This would help clarify whether the improvement comes specifically from the proposed RGB-aware masking strategy, rather than from introducing an additional edge-level constraint in general.

2 comments
AuthorsOpenReview ↗

Loss Design and Dataset Clarifications

We thank the reviewer for the detailed comments on loss attribution, masks, and dataset construction.

1. Compact-source loss attribution. Appendix C, Table 5 holds the Stage-1 checkpoint, paired data, training schedule, trainable modules, and evaluation protocol fixed. Removing the source-response group changes PSNR/SSIM/LPIPS from 24.18/0.692/0.243 to 23.10/0.652/0.281, supporting the group's contribution across all three metrics. The quantile term targets global bright-tail statistics, while Lcc penalizes underestimated intensities in selected compact bright regions. This gives complementary global and local constraints. We attribute the measured gain to their combination, rather than to Lcc alone or a particular mask construction.

2. Leakage-mask interpretation and evidence. Equation (7) selects RGB gradients at or above their own quantile threshold and IR gradients below theirs. It therefore identifies candidate texture-leakage regions using modality-relative Sobel gradient magnitudes. Weak IR structure can remain in these regions, so the mask is a heuristic rather than an absence-of-edge test. Table 5 supplies direct evidence for the term: removing it changes PSNR/SSIM/LPIPS to 23.56/0.622/0.268, compared with 24.18/0.692/0.243 for the full objective.

3. Acquisition and scene redundancy. Section 4.1 and Appendix B document shared-timestamp pairing within 20 ms and calibration-board-based spatial alignment. The shift-tolerant objective accommodates small residual translations. THIRAlign's contiguous-sequence split prevents temporally adjacent frames from crossing partitions, providing a concrete safeguard against temporal leakage.

I thank the authors for their response. Unfortunately, most of my concerns have not been adequately addressed. The additional ablation results only demonstrate the effectiveness of the combined losses, rather than the individual contribution of Lcc. Similarly, although the proposed leakage loss shows effectiveness in the new experiments, its physical justification remains unclear, and the choice of thresholds has not been sufficiently explained. Furthermore, my remaining questions have not been addressed. Therefore, I am inclined to maintain my original rating.

Reviewer_DcLwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Compared to RGB images, infrared images contain more information at night. If RGB images are converted into infrared images, infrared training data can be augmented. However, the conversion between the two is not an ordinary color transformation or style transfer. Visible light reflection and thermal radiation follow different physical processes, and paired data is relatively scarce. To address these issues, the authors propose T2IR, a two-stage conversion framework based on diffusion models. In the first stage, unpaired image sets are used to learn cross-modal appearance priors. In the second stage, the model parameters learned in the first stage are transferred and fine-tuned using paired data, with multiple regularization terms applied, including an RGB texture leakage suppression term and thermal statistics regularization term. Additionally, the authors constructed a THIRAlign dataset, including 32,214 aligned pairs of data. The authors compared T2IR with six existing methods, showing improvements in PSNR, SSIM, and LPIPS metrics on the KAIST, LLVIP, and THIRAlign datasets. Ablation experiments were also conducted to analyze the effects of the two-stage training strategy and different loss functions.

Critical strengths

1.The authors addressed the challenges of obtaining paired data, which is difficult and costly, and proposed a two-stage method that effectively combines unpaired and paired data. THIRAlign's ablation experiments also demonstrated the effectiveness of this method, with the second stage (PSNR 19.54) significantly outperforming the first stage alone (PSNR 14.32). This indicates that the two types of supervision are indeed complementary, and selectively fine-tuning only the forward path is an effective transfer strategy that also saves resources. 2.Furthermore, the authors observed that relying solely on conventional image reconstruction loss makes it difficult to adequately constrain the local thermal responses in infrared images. They proposed specialized constraints for the training phase: a high-intensity tail distribution constraint to improve the intensity distribution in bright regions, a compact heat source constraint to enhance the generation of locally concentrated heat sources, a multi-scale thermal field constraint to improve thermal imaging structures at different scales, and an RGB texture leakage constraint to reduce unwanted visible-light textures and edges. Collectively, these guide the model to generate infrared images that better match the expected appearance.

Critical weaknesses

1.Most importantly, the author cannot prove whether the generated infrared images are physically close to real infrared images. The currently used PSNR, SSIM, and LPIPS metrics can only indicate that the generated images are closer to the real targets at the image level, which does not necessarily mean that these generated images have reasonable thermal radiation characteristics. The author did not provide direct radiometric measurements for validation, nor did they compare the performance of generated infrared images with actual infrared images in downstream tasks.
2.In addition, although the loss functions in the paper reflect the author's certain design intentions, most of them rely on manually set values and require more clear validation.

Questions for authors

1 .Can you provide some evidence to prove that the generated infrared images actually enhance the physical realism of thermal imaging, rather than merely improving the image-level matching? 2 .Which of the proposed regularization terms have generalization capability? Under different infrared sensors, image resolutions, and thermal intensity distribution conditions, are these loss functions still effective, or is their effectiveness only validated under the data settings evaluated in this paper? 3.Have the authors assessed whether the generated infrared images can improve downstream visual perception tasks?

2 comments
AuthorsOpenReview ↗

Downstream Utility and Physical Limits

We thank the reviewer for distinguishing visual similarity, physical accuracy, and downstream usefulness. We agree that these require different evidence.

1. Controlled downstream evaluation. We evaluate Cityscapes-to-MFNet transfer with SegFormer-B0, batch size 4, and 100 epochs. Each generation method translates the same 2,975 Cityscapes fine-training RGB images into one IR image per input, inheriting the corresponding semantic labels. The 500 Cityscapes validation images are used only for source-domain diagnostics and are excluded from training.

The synthetic-data conditions share segmentation initialization, source-image and label lists, sample count, output dimensions, training schedule, augmentations, and checkpoint-selection rules. They use no MFNet annotations for training, model selection, or hyperparameter tuning. MFNet test images and their RGB counterparts are excluded from generator training. This controls downstream supervision and evaluation.

We use MFNet's original train.txt, val.txt, and test.txt, without random repartitioning. All methods are evaluated on the same real IR test images with identical class mapping and background/ignore handling. Every mIoU below averages exactly car, person, and bicycle, rather than the full MFNet label set. Their Cityscapes labelIds are 26/24/33, corresponding to trainIds 13/11/18 and MFNet IDs 1/2/3.

Segmentation training imagesThree-class mIoU (%) ↑
ThermalGen-generated Cityscapes IR26.38
DiffV2IR-generated Cityscapes IR24.75
TherA-generated Cityscapes IR28.43
T2IR-generated Cityscapes IR29.65
Real MFNet IR with training annotations (supervised reference)57.09

T2IR improves over ThermalGen, DiffV2IR, and TherA by 3.27, 4.90, and 1.22 percentage points, respectively. With the segmentation model and source supervision held fixed, T2IR yields the highest observed mIoU among the four synthetic-data conditions. This directly supports its downstream utility under the tested protocol, beyond image-similarity measurements alone.

The real-IR row is a target-supervised reference using the same architecture and MFNet training annotations. Its 27.44-point advantage contextualizes the remaining transfer gap; differences in annotation access and dataset size distinguish it from the controlled synthetic-data comparison.

2. Physical interpretation and transfer scope. Our contribution is thermal-image synthesis with useful intensity and structural constraints: bright-region responses, intensity statistics, and spatial structure.

Thank you for the additional evaluation and clarification of the method’s physical scope. However, the connection between the proposed regularizers and physically meaningful thermal characteristics remains insufficiently validated, and their generalization across sensors and imaging conditions remains unclear. The new results partially address my concerns, but do not sufficiently change my overall assessment.

Reviewer_aXj8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a diffusion framework for RGB to IR image translation. they propose a 2stage, RGB-IR prior learning (with unpaired image pools) followed by RGB to IR finetuning(on paired data from 9 datasets) using thermal appearance regularizers. They also propose a dataset THIRAlign, capturing RGB IR registered images of ground level and elevated road scenes. They report SOTA PSNR, SSIM, LPIPS across KAIST, LLVIP and THIRAlign.

Critical strengths

The proposed method and dataset used for training and eval settings are clear and reproduceable. Overall the method is designed well, with components to target specific rgb-thermal alignment artifacts.

Critical weaknesses

missing ablations for the need for both RGB and IR prior learning, effect of IR-500K in stage 1 compute matched stage 2 finetuning thermal specific metrics, missing evals on aerial datasets

Questions for authors

stated in the weaknesses

1 comment
AuthorsOpenReview ↗

Prior Learning and Training Fairness

We thank the reviewer for the questions about prior learning, computational fairness, and evaluation coverage. We clarify the existing comparison and provide additional downstream evidence below.

1. Prior learning versus additional training time. The Stage-2-only comparison in Table 2 already matches the total GPU-time budget. Both configurations are trained on one NVIDIA RTX 4090 for 36 hours. Full T2IR allocates 24 hours to unpaired Stage 1 and 12 hours to paired Stage 2, whereas Stage-2-only devotes all 36 hours to paired training. The paired objective, training data, trainable modules, preprocessing, and evaluation protocol are the same. Checkpoint selection does not use test performance.

ConfigurationGPU-hoursPSNR ↑SSIM ↑LPIPS ↓
Stage-2-only3619.540.5030.322
Full T2IR24 + 12 = 3624.180.6920.243

Full T2IR improves PSNR by 4.64 dB and SSIM by 0.189, and reduces LPIPS by 0.079, supporting prior learning followed by paired calibration under matched GPU time. Table 2 also isolates the benefit of the complete Stage-2 objective: compared with the matched two-stage L1+LPIPS baseline, T2IR gains 1.60 dB PSNR and 0.090 SSIM, and reduces LPIPS by 0.053. Together, these comparisons support both the training framework and the additional objective. Budget matching refers to GPU time on the same hardware.

2. Roles of the priors and IR-500K. Stage 1 learns domain-level RGB–IR appearance relationships without pair identities, while Stage 2 supplies scene-specific structural and intensity correspondence. IR-500K expands the infrared pool available for this prior learning.

Comments on the paper

3 comments

Dear Authors,

There are fewer than four days left in the author-reviewer discussion period, and you have not yet submitted a rebuttal. If you do not plan to withdraw your paper, please submit your rebuttal as soon as possible. This will give reviewers enough time to read your responses, provide feedback, and discuss the paper with you.

Thank you.

Your AC

Dear Reviewers,

There are fewer than two days left in the discussion period. The authors have submitted their rebuttal, but we have not yet received your feedback.

Please read the rebuttal and let the authors know if it answers your concerns. If you still have concerns, please share them soon so the authors have enough time to respond.

Thank you for your help.

Your AC

Dear Reviewers aXj8 and DcLw,

There are fewer than 12 hours left in the author-reviewer discussion period, and we have not yet received your feedback on the rebuttal.

Please read the authors' responses and share your feedback as soon as possible. Please let the authors know whether your concerns have been addressed. If any concerns remain, please point them out so the authors have time to respond.

Thank you for your help.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.