acceptodds
Under review as a conference paper at ICLR 2027

Frequency-Aligned Latent Adaptation for Text-to-Infrared Image Generation

Abstract

Text-to-infrared generation provides a controllable way to synthesize infrared images from text, offering a potential approach for expanding diverse infrared data resources. However, adapting pretrained text-to-image diffusion models to infrared imagery remains challenging because latent representations learned from natural images are not tailored to infrared-specific thermal structures and frequency characteristics. Directly modifying the latent space may improve infrared specialization but can disrupt the latent distribution expected by the denoising model, creating a trade-off between adaptation and latent compatibility. To address this issue, we propose Frequency-Aligned Latent Adaptation (FALA), a two-stage framework that learns infrared-aware latent corrections through frequency-aligned encoder adaptation and mean-only posterior refinement, then progressively transfers them to the denoising U-Net for stable adaptation. We further curate a large-scale infrared corpus from 69 datasets for text-to-infrared generation. Extensive experiments demonstrate that FALA consistently improves distribution fidelity and coverage across diverse infrared scenarios while preserving the pretrained diffusion prior without additional inference-time overhead.

Primary area:
generative models
Keywords:
Text-to-Infrared Generation, Diffusion Models, Generative Model Adaptation

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_7q8EOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a two-stage latent representation adaptation framework for text-to-infrared image generation. In the first stage, adapters with low- and high-frequency branches are introduced into a frozen pretrained VAE backbone to model global thermal structures and local details, respectively. A latent mean refinement module adjusts the posterior mean while preserving the original posterior variance. In the second stage, progressive interpolation between the original and adapted latent means enables the denoising U-Net to gradually adapt to infrared-specific latent representations. The authors also curate an infrared image–text dataset comprising approximately 1.108 million images from 69 data sources.

Critical strengths

1.The adaptation framework is practical and does not introduce additional inference-time modules. 2.The dataset curation and experimental evaluation constitute useful contributions.

Critical weaknesses

1.The related work section could provide a more comprehensive overview of recent advances in infrared vision. The authors are encouraged to discuss additional representative studies, such as: [1]Symbiotic Evolutionary Learning for Task-Adaptive Infrared and Visible Image Fusion,2026,TPAMI. 2. The paper contains several minor presentation issues, including inconsistent capitalization in BibTeX entries, spelling errors, and inconsistent notation for adaptation strength. The authors should also remove residual editing notes from the references to improve overall clarity and consistency.

Questions for authors

1.What is the main methodological advance over existing latent representation adaptation and frequency modeling approaches? What role does each component play, and why is it necessary to combine them? 2.When the posterior variance is held fixed, how do distributional changes caused by adjusting the posterior mean affect the pretrained model? How do the proposed constraints mitigate these effects, and could the authors provide supporting analyses? 3.Under matched training conditions, could the authors provide generation ablations for the key modules and different posterior variance settings to determine whether improvements in reconstruction and frequency consistency also translate into better final generation quality?

Reviewer_2BGeOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper focuses on text-to-infrared image generation and, in particular, on how the latent representation is adapted for infrared data. Instead of only fine-tuning the denoising network, the authors argue that the representation gap between visible and infrared images should also be considered. At the same time, changing the latent space too much may reduce its compatibility with the pretrained diffusion model. To address this issue, the authors propose FALA, which is trained in two stages. The first stage adapts the frozen VAE encoder using multi-scale frequency-aware modules and further refines the posterior mean while retaining the original variance. The second stage gradually transfers the adapted infrared representation to U-Net training rather than introducing it all at once. In addition, the authors collect infrared images from 69 public datasets and sources to form a relatively large training corpus. Experiments include comparisons on several generation metrics, results across different scenes, qualitative examples, and ablation studies. Overall, FALA performs better than the considered adaptation baselines in the reported experiments without changing the standard inference procedure of latent diffusion.

Critical strengths
  1. The paper has a clear technical motivation. Rather than adapting only the denoising network, it explicitly considers how the infrared latent representation itself should be changed while still remaining usable by the pretrained diffusion model. The combination of frequency-aware encoder adaptation, mean-only posterior refinement, and progressive latent transition forms a coherent pipeline rather than a collection of unrelated modules. The method is also practical in that the additional latent adaptation components are used only during training, while inference retains the standard SD1.5 latent-diffusion pipeline without extra modules or denoising steps.

  2. The experimental evidence is fairly consistent. FALA performs well in the overall Test10K comparison and the improvements extend across several infrared scenarios. The paper also includes targeted ablations for the frequency decomposition, branch fusion, global context modeling, latent mean refinement, and progressive latent adaptation, which makes it easier to understand where the gains come from.

Critical weaknesses
  1. Some Stage-I training details could be reported more explicitly, especially the learning rate and the coefficients used for the different loss terms. These details would make the training setup easier to reproduce and would clarify how the different objectives are balanced.

  2. FALA itself is evaluated on SD1.5. The experiments are sufficient to support the claims made in the current setting, but it is still unclear whether the same adaptation strategy would transfer equally well to other latent diffusion backbones or latent representations. This mainly limits the scope of the current empirical conclusions.

Questions for authors
  1. The low-frequency branch combines bidirectional recurrent aggregation with selective scanning. Could the authors comment a bit more on what each component contributes, based on the GRU-only and scan-only results?

  2. The dataset split groups related samples rather than performing image-level random splitting. Could the authors briefly clarify how this procedure handles video sequences and sources where reliable sequence information is unavailable?

  3. For the baseline comparison, could the authors clarify which settings are kept identical across methods, especially the training corpus, prompts, output resolution, and random seeds?

  4. The paper states that FALA does not introduce additional inference-time overhead. Could the authors clarify which components are used only during training and what exactly is retained during inference?

Reviewer_brpXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes text to ir generation with diffusion model adpatation.

Critical strengths

Its performance is improved from baseline.

Critical weaknesses

The model itself is improved from baseline and the overall method is plausible as it adapts SD model for different domain generation.

However I have major concerns whether it is really state of the art method which can be eligible for published in 2027.

All of the baselines and comparisons are limited to models published in 2024 (which are limited to SD1.5, SDXL, Pixart) . I don’t follow the recent technical updates on this field but there would be definitely better approach different from conventional Unet based model.

I think even simple lora fine tuning on recent pretrained generative model such as Qwen image . Please mention or compare these recent DiT based flow models to genuinely prove the models superior performance

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.