acceptodds
Withdrawn from ICLR 2027

Thermo-VL: Extending Vision–Language Models to Thermal Infrared Perception

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) often fail under low illumination because their visual grounding is learned predominantly from RGB imagery, whereas thermal infrared preserves complementary scene structure when visible cues degrade. We present **Thermo-VL**, a language-conditioned RGB-thermal VLM that augments a frozen Molmo-7B backbone with a trainable thermal encoder and a text-guided dual-attention fusion module. Given aligned RGB tokens, thermal tokens, and prompt embeddings, the fusion module conditions thermal features on both language and RGB context, then injects a gated residual into the frozen RGB stream so thermal evidence can be incorporated without disrupting Molmo's pretrained RGB-language interface. We train the model with the standard language-modeling objective together with auxiliary alignment and regularization losses that improve cross-modal grounding and reduce over-reliance on RGB. We also introduce a pixel-aligned RGB-thermal instruction-tuning dataset and Thermo-VL-Bench, a manually screened RGB-thermal VQA benchmark for low-light and cross-spectrum reasoning. Experiments show improvements in thermal-only reasoning and substantial gains in RGB+thermal reasoning, highlighting the value of prompt-conditioned RGB-thermal fusion. Data and code will be made publicly available after the review process.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language models, multispectral learning, thermal infrared imaging, RGB-thermal fusion, low-light visual reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nF2GOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

To address the failure of existing VLMs under low-light conditions, the method is built on the existing Molmo-7B backbone. At the image-encoding truncation point, it enhances the encoded visual tokens through a trainable thermal encoder and a text-guided dual-attention fusion module. It also introduces a new dataset, ThermoVL-Bench.

Critical strengths

There is some experimental support and a certain performance advantage.

Critical weaknesses

The method is essentially an early-fusion method. It achieves enhancement by fusing information from different modalities at the input end.

This design is straightforward, but the authors lack generalization studies. Some classic VLMs, such as Qwen-VL and LLaVA, have not been evaluated. There are too few experiments.

The proposed dataset lacks analysis. It was generated using OpenAI GPT-5-mini, and the quality is too poor. Why was a better AI not used? At the same time, how was the AI-generated content verified to ensure the quality of the dataset itself? This part is missing from the paper. There is too much empty space in the nine-page main text. The quality of the figures is too poor, and the paper as a whole lacks refinement.

Questions for authors

The quality is too poor.

Reviewer_tTBEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Thermo-VL, an RGB–thermal vision-language model built on a frozen Molmo-7B-D backbone. It adds a partially trainable thermal ViT encoder and a fusion module in which thermal tokens attend to both the user prompt and RGB tokens before being injected into the RGB stream through a token-wise gated residual. The paper also introduces an instruction-tuning dataset containing approximately 19.5K RGB–thermal pairs and 54K QA pairs, together with Thermo-VL-Bench, which contains approximately 15K binary questions over 3,148 real RGB–thermal pairs. Thermo-VL achieves 88.96% RGB+IR accuracy on the proposed benchmark and 73.20% overall accuracy on RGB-Th-Bench.

Critical strengths

S1. Potentially Useful Data Contribution

  • The paper introduces a relatively large RGB–thermal dataset that supports both instruction tuning and evaluation. It contains substantially more image pairs than RGB-Th-Bench. The authors also take several useful quality-control steps, including sequence-level splitting, duplicate checking, manual screening, and an independent annotation audit. They clearly disclose that the questions are model-generated and the RGB–thermal images are assumed to be aligned. But, the thermal images do not provide calibrated temperature values.
Critical weaknesses

W1. Insufficient Comparison with Prior Work

  • The related-work section misses several highly relevant recent studies [1-6]. For instance, DualVision (Majeedi et al., CVPR Findings, 2026) already builds an RGB–IR MLLM using aligned image pairs, cross-attention fusion, a large RGB–IR instruction dataset, and a benchmark for multimodal reasoning under visual degradations. Its 25K paired images and 204K QA annotations are directly comparable in purpose to the resources proposed here. The primary architectural distinction appears to be Thermo-VL’s additional prompt-conditioned attention and token-wise gate, but this difference is neither discussed nor evaluated experimentally.

  • The positioning of earlier infrared VLMs is also incomplete. Infrared-LLaVA directly trains an infrared-language model using infrared-specific supervision, while IRGPT introduces more than 260K real infrared-text samples and nine infrared-language tasks. Therefore, the statement that previous methods do not directly align thermal observations with language is too broad. Infrared-LLaVA (Jiang et al., Findings of EMNLP, 2024) and IRGPT (Cao et al., ICCV, 2025) should be discussed more accurately.

  • Overall, the methodological contribution currently appears to be an incremental variation of established cross-attention, residual adaptation, modality alignment, and gating mechanisms applied to an RGB–thermal VLM.

[1] DualVision: RGB-Infrared Multimodal Large Language Models for Robust Visual Reasoning, CVPR 26 Findings

[2] IRGPT: Understanding Real-world Infrared Image with Bi-cross-modal Curriculum on Large-scale Benchmark, ICCV 25

[3] Infrared-LLaVA: Enhancing Understanding of Infrared Images in Multi-Modal Large Language Models, EMNLP Findings 24

[4] Enhanced Vision-Language Models for Diverse Sensor Understanding: Cost-Efficient Optimization and Benchmarking, TIP 26

[5] IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting, CVPR 26

[6] ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery, KDD 26

W2. Unfair Comparison Setting

  • The main comparison in Table 2 is not sufficient to isolate the proposed architecture. Thermo-VL is trained on approximately 54K domain-specific QA pairs, whereas InternVL, Qwen2.5-VL, LLaVA-Next, and Molmo appear to be evaluated out of the box. Consequently, these results combine the effects of thermal instruction tuning, thermal encoder pretraining, additional trainable capacity, and the proposed fusion design.

W3. Benchmark Reliability & Diversity

  • Thermo-VL-Bench is described as manually screened, but its questions and answers were generated by GPT-5.1 and approximately 80% were accepted verbatim. Only 100 of roughly 15K questions received independent overlapping annotation. Also, the benchmark contains only binary yes/no questions. This measures recognition and constrained verification more directly than open-ended thermal reasoning. Recent benchmarks such as IF-Bench and ThermEval provide broader task structures, while ThermEval explicitly includes quantitative temperature-grounded classification and regression. The paper should include human-authored questions, paraphrase/template-held-out evaluation, text-only and image-permutation controls, etc.

W4. Unsupported Selective-Gating Claim

  • The paper claims that the model injects thermal evidence only where it is useful for the current question. However, the experiments mainly show that thermal information is used: zeroing or shuffling thermal tokens reduces accuracy, and removing text attention also reduces performance. These experiments do not demonstrate that the learned token gates are spatially selective or change appropriately with the question.
Questions for authors

Q1. Novelty relative to recent RGB–thermal VLMs

  • How does Thermo-VL differ technically from related works [1-6], which already combines aligned RGB–IR inputs, cross-attention fusion, instruction tuning, and multimodal reasoning? Can the authors compare the two fusion methods under the same backbone, training data, and evaluation protocol?

Q2. Positioning against infrared-language models and benchmarks

  • The paper states that previous methods do not directly align thermal observations with language. How is this claim consistent with Infrared-LLaVA [3] and IRGPT [2], which explicitly train infrared-language models? Can the authors also evaluate Thermo-VL on IF-Bench [5] or ThermEval [6] to demonstrate broader thermal understanding?

Q3. Fair comparison under matched thermal instruction tuning

  • Can InternVL, Qwen2.5-VL, LLaVA-Next, and Molmo be adapted using the same 54K QA pairs and a comparable number of trainable parameters? Without such controls, how much of Thermo-VL’s improvement comes from the proposed fusion design rather than thermal instruction tuning, thermal pretraining, or additional capacity?

Q4. Possible question-template and language bias

  • Because GPT-5-mini generates the training questions and GPT-5.1 generates the evaluation questions, how do the authors rule out shared question templates or linguistic patterns? Please report results using human-authored questions, held-out question templates, paraphrased questions, a text-only baseline, and shuffled or mismatched images.

Q5. Evidence for question-dependent selective gating

  • Can the authors visualize the token-wise gates and show that their spatial distributions change when different questions are asked about the same RGB–thermal pair? Prompt-swapping experiments, gate statistics by question type, and comparisons with fixed or uniform gates would more directly support the selective-gating claim.
Reviewer_6RLeOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper extends a frozen VLM to thermal infrared. A trainable thermal encoder feeds a lightweight fusion module that conditions thermal features on the question and the RGB image, then adds them back into the RGB tokens through a gated residual, so the pretrained backbone stays untouched. The authors also contribute an RGB-thermal instruction-tuning set, built from public paired datasets and synthetic thermal images with LLM-generated QA, and Thermo-VL-Bench, a manually screened yes/no benchmark split by the modality each question needs. Thermo-VL improves substantially over the base model when both modalities are available, on Thermo-VL-Bench and on the external RGB-Th Bench.

Critical strengths
  1. The paper addresses RGB-thermal fusion, which is an important problem that has received less attention within the Vision-Language Modeling domain.
  2. The fusion design is simple and cheap to train, and the experiments show that it uses thermal evidence. It keeps the pretrained model frozen, and the experiments confirm that conditioning on Molmo's own prompt embeddings works better than using a separate CLIP text encoder. Thermo-VL beats a simpler fusion baseline trained the same way, its accuracy drops when the thermal input is removed or shuffled, and the gains carry over to an external benchmark.
  3. The dataset and benchmark would be helpful to the community. The benchmark is larger than existing RGB-thermal VQA benchmarks, and I appreciate that human reviewers screened the candidate questions and that an independent audit showed high inter-annotator agreement.
Critical weaknesses
  1. The baselines do not fully isolate what the fusion design adds. The only trained fusion baseline is a residual MLP over concatenated features. The paper never fine-tunes the standard alternative of appending thermal tokens to the RGB tokens, so its claim that fusing inside the model beats appending at the input rests on untrained baselines. An RGB-only fine-tuned baseline is also missing, so it is unclear how much of the gain comes from in-domain fine-tuning alone.
  2. The results does not yet support the paper's claims about low light and selective injection. Low light motivates the work, but the results are not broken down by lighting or tested under RGB degradation, and no gate analysis shows that thermal evidence is injected only when useful.
  3. The related work section misses several closely related papers. While experimental comparisons may not be necessary, but at least discussing these works and how the method differs from them would enrich that section:
    • DualVision (CVPR 2026 Findings) also fuses RGB+IR into a frozen MLLM with a lightweight module and releases an IR-RGB instruction-tuning dataset and benchmark.
    • IRGPT (ICCV 2025) builds an MLLM on large-scale real infrared data and argues against synthetic infrared, which bears on the paper's synthetic thermal data.
    • ImageBind (CVPR 2023) aligns thermal images with images and text in a shared embedding space, similar to the paper's contrastive objective, and LLMs built on it already accept thermal input.
Questions for authors
  1. I would be interested to see how Thermo-VL compares with Molmo fine-tuned on the same data and protocol with RGB only, a fine-tuned token-concatenation baseline (thermal tokens appended to the RGB tokens), and a text-only baseline. A subset such as just the concatenation baseline would be the most informative.
  2. I would be interested in accuracy split by day and night, or by illumination level. Since low light motivates the work, this would show where the thermal stream helps most.
  3. I would also appreciate gate statistics or visualizations across question types and lighting conditions, to see how the gate values vary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.