acceptodds
Under review as a conference paper at ICLR 2027

ThermOV: Thermal-Only Open-Vocabulary Semantic Segmentation via Unsupervised Domain Adaptation

Abstract

Open-vocabulary semantic segmentation (OVSS) enables pixel-level recognition of arbitrary visual concepts specified by text. However, existing OVSS methods are predominantly developed for RGB imagery, while prior studies involving thermal data typically rely on co-registered RGB-thermal inputs. As a result, open-vocabulary segmentation from thermal observations alone remains largely unexplored. To bridge this gap, we propose ThermOV, a two-stage unsupervised cross-domain adaptation framework that transfers the open-vocabulary segmentation capability of the RGB-trained foundation model SAM3 to the thermal domain without requiring pixel-level annotations of real thermal images. In the first stage, labeled RGB images are used to generate synthetic thermal images to provide an initial adaptation to thermal appearance. In the second stage, the model is further adapted using unlabeled real thermal images within an EMA teacher-student framework. To preserve open-vocabulary generalization during this process, we construct a diverse prompt bank that excludes categories overlapping with the target evaluation classes and jointly optimize pseudo-label supervision, semantic prompt consistency and teacher-student prediction consistency. Experiments on FMB and TIR-SS show that ThermOV improves over SAM3 by 5.97 and 9.30 mIoU, respectively, and outperforms the strongest RGB OVSS baseline adapted using labeled real thermal data by 3.81 and 0.54 mIoU. These results demonstrate the effectiveness of ThermOV for thermal-only OVSS.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
thermal-only semantic segmentation, open-vocabulary, unsupervised domain adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_khAwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes ThermOV, a two-stage framework that adapts the RGB-based SAM3 model for thermal-only open-vocabulary semantic segmentation (OVSS) without requiring pixel-level annotations of real thermal images. Stage I generates synthetic thermal images from COCO-Stuff and adapts SAM3's visual encoder using LoRA. Stage II uses unlabeled KAIST thermal images with pseudo-label supervision, EMA teacher–student consistency, and semantic prompt consistency. The method is evaluated on FMB and TIR-SS using 16 evaluation concepts with three text prompts per concept. Compared with the original SAM3, ThermOV improves mIoU on FMB and TIR-SS. The paper also provides component ablations and loss-weight sensitivity analysis.

Critical strengths
  1. The paper addresses a meaningful practical problem by enabling thermal-only open-vocabulary segmentation without pixel-level annotations of real thermal images, and reports promising improvements over SAM3 (+5.97 mIoU on FMB and +9.30 on TIR-SS), supported by qualitative examples.

  2. The component ablations in Table 3 and the prompt robustness and loss-weight analyses in Table 2 and Table S8 provide useful evidence about the contributions of the two training stages and the different consistency objectives.

Critical weaknesses
  1. The claimed separation between training and evaluation concepts is inconsistent with the appendix, as Section 4.1 states that the 76 training concepts and 16 evaluation concepts are semantically disjoint, while Tables S4 and S5 contain overlapping concepts such as road, building, and vehicle; furthermore, the paper does not clearly explain how evaluation concepts were excluded from KAIST training images or how class-specific thermal priors were handled during Stage I, making it difficult to verify the claimed unseen-concept generalization, although this does not necessarily prove actual data leakage.

  2. Several performance claims and experimental details need clarification, as the reported +3.81 mIoU improvement over the strongest supervised baseline on FMB does not match Table 1 (the actual difference is +7.45), EBSeg achieves higher mIoU than ThermOV on TIR-SS (62.50 vs. 62.14), and the baselines use different adaptation data (labeled SODA versus unlabeled KAIST); additionally, the strong loss-weight sensitivity in Table S8 and the lack of a clearly described validation and model-selection protocol make the reliability of the reported results difficult to assess.

Questions for authors

How were KAIST images containing evaluation concepts identified and excluded when selecting the 7,493 training images? Were ground-truth annotations, metadata, or automatic filtering used? Also, how were class-specific thermal priors applied in Stage I while excluding evaluation concepts from training?

Reviewer_M8ohOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents ThermOV, a two-stage framework for thermal-only open-vocabulary semantic segmentation. In Stage I, labeled RGB images from COCO-Stuff are converted into pseudo-thermal images using grayscale conversion, category-dependent intensity offsets, and image perturbations. LoRA adapters are then used to adapt the SAM3 visual encoder. In Stage II, the adapted model generates pseudo-labels for unlabeled KAIST thermal images, followed by EMA teacher–student training with pseudo-label, prediction-consistency, and prompt-consistency losses. The method is evaluated on thermal subsets of FMB and TIR-SS using 16 evaluation concepts and three prompt styles.

Critical strengths

S1. [Task perspective] Thermal-only Open-vocabulary Segmentation

  • Thermal-only open-vocabulary segmentation is useful in nighttime, low-light, and privacy-sensitive environments where RGB images may be unavailable or unreliable. The paper studies a practically relevant problem that has received less attention than RGB-T segmentation.
Critical weaknesses

W1. Incomplete Positioning Against Thermal and RGB-T Domain Adaptation

  • The related-work discussion does not adequately cover the extensive literature on unsupervised domain adaptation for thermal and RGB-T semantic segmentation. The paper briefly mentions Chen et al. (2022) and Gan et al. (2023), but omits several more direct precedents.

  • HeatNet already introduced a two-stage teacher-student framework with cross-domain feature alignment for transferring daytime RGB knowledge to nighttime RGB-T segmentation. MS-UDA subsequently proposed successive RGB-to-RGB, RGB-to-thermal, and thermal-to-thermal adaptation. It used predictions from an RGB segmentation model as pseudo-labels and CycleGAN-generated nighttime thermal images to reduce both spectral and day-night domain gaps. Thus, the combination of synthetic thermal data, pseudo-label transfer, and staged adaptation was already explored in thermal segmentation by 2021.

  • More recent work has further developed this direction. Gan et al. perform RGB-to-thermal adaptation without thermal annotations or registered RGB-T pairs. Night-TTA employs cross-modal ensemble distillation for nighttime adaptation. Kwon et al. introduce masked mutual learning, cross-spectral prototypes, and self-supervised pseudo-labeling while retaining thermal-only inference. UMDA similarly combines day-to-night generation, pseudo-label supervision, and cross-modal self-distillation.

  • The paper should position ThermOV against this progression rather than treating thermal UDA mainly as conventional fixed-category adaptation. Although these methods are not open-vocabulary methods, they are the closest methodological precedents for nearly every adaptation component used by ThermOV.

[1] Vertens, Zürn, and Burgard, “HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images,” IROS, 2020.

[2] Kim, Shin, Park, and Kweon, “MS-UDA: Multi-Spectral Unsupervised Domain Adaptation for Thermal Image Semantic Segmentation,” IEEE RA-L/IROS, 2021.

[3] Chen et al., “Light Transport Induced Domain Adaptation for Semantic Segmentation in Thermal Infrared Urban Scenes,” IEEE T-ITS, 2022.

[4] Gan, Lee, and Chung, “Unsupervised RGB-to-Thermal Domain Adaptation via Multi-Domain Attention Network,” ICRA, 2023.

[5] Liu et al., “Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentation,” IEEE TAI, 2024.

[6] Kwon et al., “Boosting Cross-Spectral Unsupervised Domain Adaptation for Thermal Semantic Segmentation,” ICRA, 2025.

[7] Chen et al., “Unsupervised Multi-Modal Domain Adaptation for RGB-T Semantic Segmentation,” CVIU, 2026.

W2. Limited Methodological Novelty Beyond Standard UDA and Consistency Learning

  • The proposed training strategy largely combines standard input-level and output-level domain-adaptation techniques. At the input level, Stage I converts RGB images to grayscale, adds manually defined category-dependent intensity offsets, and applies conventional image perturbations. Earlier thermal UDA methods already used synthetic thermal or day-to-night thermal generation to reduce the input-domain gap. Compared with learned translation or physically motivated thermal generation, the proposed transformation is considerably simpler and is not validated against real thermal-image statistics.

  • At the output level, Stage II first generates pseudo-labels on unlabeled target images and then trains an EMA teacher-student model using differently augmented views. This is a standard self-training and consistency-learning structure. Mean Teacher introduced EMA-based teacher targets, FixMatch combined weakly augmented pseudo-label generation with strongly augmented student training, and MIC and UniMatch extended similar consistency principles to domain adaptation and semantic segmentation. Therefore, the main novelty appears to be applying a conventional UDA pipeline to a SAM3-based thermal-only OVSS setting.

[8] Tarvainen and Valpola, “Mean Teachers Are Better Role Models,” NeurIPS, 2017.

[9] Sohn et al., “FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence,” NeurIPS, 2020.

[10] Hoyer et al., “MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation,” CVPR, 2023.

[11] Yang et al., “Revisiting Weak-to-Strong Consistency in Semi-Supervised Semantic Segmentation,” CVPR, 2023.

W3. Unconvincing Benchmark Performance

  • The empirical results are not strong enough to support a broad state-of-the-art claim. ThermOV clearly improves over the unadapted SAM3, but it is not consistently the best method even in Table 1. On TIR-SS, EBSeg† reaches 62.50 mIoU, which is higher than ThermOV’s 62.14. EBSeg† also achieves 76.10 mAcc, substantially above ThermOV’s 68.80. The claimed 0.54-mIoU improvement is obtained only relative to Mask-Adapter† at 61.60, which is not the strongest baseline. Therefore, the paper does not establish consistent superiority over supervised thermal adaptation.

  • The improvement also cannot be clearly attributed to a proposed method. Stage-I pseudo-thermal adaptation alone improves SAM3 by only 0.72 mIoU on FMB and 2.55 on TIR-SS. Most of the gain comes from conventional target-domain pseudo-label training and teacher-student consistency. Without same-backbone UDA baselines, it is unclear whether ThermOV performs better than simply applying an established self-training method to SAM3.

W4. Weak Validation of the Pseudo-Thermal Generator

  • The main components, such as LoRA adaptation, pseudo-label training, EMA teacher-student consistency, and prompt consistency, are established techniques. The main technical distinction is their application to thermal-only OVSS.

  • Moreover, the pseudo-thermal generator is based on grayscale conversion and manually selected class-dependent brightness offsets. Thermal appearance depends on temperature, material emissivity, weather, sensor response, automatic gain control, and imaging polarity. The paper provides no quantitative or perceptual evidence that the generated images resemble real thermal data. There is also no ablation separating grayscale conversion, category priors, and appearance perturbations.

W5. Limited Reliability & Reproducibility Reporting

  • Several implementation details are also unclear, including the pseudo-label confidence threshold, candidate-mask merging, overlap resolution, background assignment, and final multiclass prediction procedure.
Questions for authors

Q1. Novelty relative to existing thermal UDA methods

  • What is technically new in ThermOV compared with HeatNet, MS-UDA, Boosting Cross-Spectral UDA, and other recent UDA methods, which already use staged cross-spectral adaptation, synthetic thermal data, pseudo-label transfer, and teacher–student or self-supervised learning? Is the main contribution a new adaptation method, or the application of established UDA techniques to SAM3-based thermal-only OVSS?

Q2. Validity of the pseudo-thermal generator

  • Can the authors quantitatively compare the generated pseudo-thermal images with real thermal images using intensity distributions, feature-space distances, or other thermal-domain measures? Please also compare the proposed handcrafted transformation with learned thermal translation and provide separate ablations for grayscale conversion, category-dependent thermal priors, and appearance perturbations.

Q3. Interpretation of the benchmark results

  • How is the claimed 0.54-mIoU improvement over the strongest supervised thermal-adaptation baseline calculated? In Table 1, EBSeg† achieves 62.50 mIoU on TIR-SS, exceeding ThermOV’s 62.14, while also obtaining substantially higher mAcc. Please correct the claim and clarify under which metric and baseline ThermOV is considered superior.

Q4. Source of the reported performance gain

  • Can the authors provide same-backbone comparisons with SAM3 plus generic target-domain self-training and SAM3 adapted using labeled SODA data? Given that Stage I alone provides only +0.72 mIoU on FMB and +2.55 on TIR-SS, how much of the final improvement comes from the proposed method rather than standard pseudo-label training on unlabeled thermal images?

Q5. Complete pseudo-labeling and inference procedure

  • Can the authors report the pseudo-label confidence threshold, confidence calibration procedure, candidate-mask filtering and merging rules, treatment of overlapping masks, background assignment, and the procedure for converting prompt-conditioned binary masks into the final multiclass prediction? These details are necessary to reproduce both training and evaluation.

Q6. Training and Evaluation Concept Overlap

  • The paper states "we construct three prompts for each concept: a canonical category name, a synonymous expression, and a descriptive phrase. This results in 48 evaluation prompts, which are semantically disjoint from the 76 concepts represented by 228 prompts in our training prompt bank."

  • However, Table S4 lists “road,” “building,” and “vehicle” as training concepts, while Table S5 uses exactly the same concepts for evaluation. “Automobile” and “building structure” also appear in both prompt banks. How can the training and evaluation vocabularies be semantically disjoint when “road,” “building,” and “vehicle” appear in both Tables S4 and S5? Please provide the complete concept and prompt lists.

Reviewer_j7V5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a Open vocabulary semantic segmentation adaptation framework, using synthetic thermal data from labelled RGB images and then a unsupervised adaptation to real thermal images.

Critical strengths

the application of Open vocabulary semantic search on thermal images is itself limited in exploration. the overall findings support the claim of the paper that both stages of the pipeline help to give an overall aggregated boost. the evaluation shoes improvement over the baseline model SAM3.

Critical weaknesses

The reported benchmarks arent consistent across different tables. eg: Sec 4.3, the adapted Mask Adapter in Table 1, against the canonical results of table 2, on TiR-SS the EBSeg remains SOTA across mIOU and mAcc

missing ablations on stage 1 SAM3-PTA finetuning, why the different choices like category aware thermal prior, pertubations.

Other representative methods use much smaller ViTB based baselines, while sam3 remains about 3x larger, the zero shot performance gives of sam3 gives a unfair advantage.

Questions for authors
  • Adding details to why the design choices were made for the first stage
  • address the reported metrics issues in table 1 and 2
  • Justify the performance boost wrt params and baseline performance. how well does this method generalise to other model architectures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.