acceptodds
Withdrawn from ICLR 2027

Beyond Unidirectional Prior Injection: Bidirectional Semantic Alignment for RGB-T Semantic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

RGB-T semantic segmentation benefits from the complementary information of visible and thermal images, while pre-trained large-scale vision models (LVMs) provide rich semantic priors for scene understanding. However, transferring RGB-pretrained semantics to RGB-T representations is nontrivial because the pre-trained semantic space is optimized for visible-image distributions and may be mismatched with heterogeneous multimodal features. To address this issue, we propose a Bidirectional Semantic Alignment Network (BiSANet), which enables target-aware interaction between RGB-T representations and pre-trained semantic priors. A Hierarchical Dual-Modal Encoder first constructs stable and complementary RGB-T features. Bidirectional Semantic Adapters then inject target-domain information into the frozen semantic pathway and transfer the adapted priors back to the multimodal branch for semantic correction. Finally, a Semantic Enhancement Decoder integrates multi-level features for fine-grained prediction. BiSANet achieves state-of-the-art performance on MFNet, PST900, and FMB, demonstrating effective semantic-prior adaptation for RGB-T segmentation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RGB-T semantic segmentation, semantic priors, bidirectional semantic alignment, multimodal fusion, Large-scale Vision Models (LVMs)

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mp2eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BiSANet for RGB‑T semantic segmentation, which builds bidirectional semantic alignment between RGB‑T multimodal features and frozen SAM pre‑trained semantic priors. The model consists of three core modules: Hierarchical Dual‑Modal Encoder (HDE), Bidirectional Semantic Adapter (BSA), and Semantic Enhancement Decoder (SED). HDE implements level‑wise cross‑modal fusion for RGB‑T inputs. BSA contains Injector and Extractor components: Injector injects RGB‑T target‑domain information into SAM transformer blocks to adapt pre‑trained semantics, and Extractor transfers adapted SAM priors back to calibrate multimodal representations. SED fuses multi‑level features together with auxiliary loss heads for foreground‑background and boundary supervision.

Critical strengths
  1. The research motivation is clear: existing foundation‑model‑driven RGB‑T segmentation mainly adopts unidirectional prior injection, ignoring domain adaptation of pre‑trained RGB semantic priors toward thermal modality. The bidirectional interaction perspective is reasonable.
  2. Comprehensive benchmark evaluations are performed on three mainstream RGB‑T semantic segmentation datasets, with rich ablation studies covering module components, adapter variants, loss combinations, and encoder efficiency trade‑offs. Visualization analyses including t‑SNE, CKA, MMD, weight visualization and residual response maps are provided to interpret model behaviours.
  3. The hierarchical fusion design inside HDE considers different characteristics of low‑level spatial details and high‑level semantic features for RGB‑T data. Auxiliary supervision for boundary and foreground‑background further improves segmentation boundary quality.
Critical weaknesses
  1. Limited core novelty. The bidirectional adapter is built upon vanilla cross‑attention. The injector and Extractor are essentially two independent cross‑attention layers without special architectural innovation. Similar bidirectional cross‑modal interaction patterns have been widely explored in prior RGB‑T segmentation and multimodal foundation‑model adaptation works. The main gain partially comes from the heavily frozen SAM ViT‑L backbone rather than purely from the proposed bidirectional alignment mechanism.
  2. Insufficient baseline comparison for foundation‑model setting. The paper lacks fair baselines that adopt the same frozen SAM backbone with only unidirectional adapter modifications. The current baseline uses simple MFNet‑style fusion, which is not a strong representative of existing SAM‑based RGB‑T methods. It cannot solidly prove that performance improvement originates from bidirectional design instead of stronger fusion or decoder.
  3. Ambiguity on gain attribution. Table 5 ablation shows that standalone BSA brings marginal improvement only (+0.5 mIoU). The bidirectional adapter achieves large performance boost only when combined with HDE. It is hard to tell how much improvement is genuinely contributed by bidirectional semantic alignment itself. Furthermore, the nighttime gain shown in Table 9 is promising, but the paper lacks a quantitative statistical significance test to verify whether such improvement is stable and non‑random.
  4. Efficiency and practical concerns. When equipped with SAM ViT‑L, the model only runs at 11.4 FPS with 556.4 G FLOPs, which severely limits real‑world deployment for autonomous driving and surveillance scenarios targeted by RGB‑T segmentation. Although ViT‑B / ViT‑T lightweight variants are provided, noticeable performance drop exists. In addition, the paper does not conduct inference‑time complexity analysis for the proposed adapter modules.
  5. Missing critical analysis on failure cases. The failure case discussion is superficial. The authors only mention extremely small‑object and low‑contrast failure modes, yet do not provide actionable insight for how bidirectional alignment fails under these challenging conditions. There is no discussion about failure difference between unidirectional versus bidirectional adapters on hard samples.
  6. Writing drawback. Some descriptions are vague. For instance, the design motivation for adapter insertion positions (11,17,23 blocks of SAM) lacks sufficient theoretical justification, only supported by ablation results.
Questions for authors
  1. In Table 5, standalone BSA only brings marginal mIoU improvement (+0.5%). The large performance gain emerges only when BSA is combined with HDE. Could you quantitatively decompose and report the isolated performance contribution purely from your bidirectional alignment mechanism (Injector+Extractor), under a fixed, strong multimodal fusion baseline that is already equipped with frozen SAM? The current baseline adopts simple MFNet‑style fusion, which cannot rule out that gains come from improved fusion rather than bidirectional cross‑attention.
  2. Your key claim is that bidirectional adaptation yields large improvements especially under night / low‑light conditions (Table 9). Are the nighttime mIoU improvements of +3.0% (vs Extractor‑only) statistically significant? Please provide paired statistical test results (e.g., paired t‑test across test splits / multiple repeated runs with different random seeds) to exclude the possibility that this observed gain arises from random training fluctuation.
  3. Why are SAM blocks 11, 17, 23 selected as insertion positions for Bidirectional Semantic Adapters besides ablation results? Is there any architectural or theoretical motivation, other than brute‑force ablation over candidate transformer indices?
  4. What is the exact additiona/ latency) introduced only by the Injector‑Extractor adapter modules, decoupled from the heavy SAM ViT‑L backbone? It is difficult to judge whether the adapter itself introduces substantial inference cost.
  5. In failure‑case analysis, for those hard samples with tiny, low‑contrast objects: quantitatively compare prediction errors (mIoU on failure subset) between unidirectional variants (Injector‑only / Extractor‑only) and full bidirectional BSA. Does bidirectional alignment still bring a consistent advantage on these challenging samples, or does it fail equally as unidirectional prior injection?
Reviewer_1ohTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents BiSANet, an RGB-thermal semantic segmentation framework that integrates a frozen Segment Anything Model (SAM) with a trainable dual-modal segmentation branch. A Hierarchical Dual-Modal Encoder (HDE) first combines RGB and thermal features using low-level difference-aware fusion and high-level adaptive weighting. Bidirectional Semantic Adapters (BSAs) then inject RGB-T features into intermediate SAM blocks and extract the adapted SAM features back into the RGB-T branch. A Semantic Enhancement Decoder (SED) combines multi-level features using SegFormer-style projections, ASPP, and auxiliary foreground/background and boundary prediction heads. The method reports 61.2%, 88.2%, and 67.9% mIoU on MFNet, PST900, and FMB, respectively. The results are competitive, particularly under nighttime conditions. However, the architectural novelty, state-of-the-art claim, comparison fairness, and evidence for semantic alignment are not sufficiently established.

Critical strengths

None

Critical weaknesses

W1. Limited Architectural Novelty

  • The central Bidirectional Semantic Adapter is very similar in structure and terminology to the injector-extractor interaction introduced in Vision Transformer Adapter for Dense Predictions (Chen et al., ICLR 2023). ViT-Adapter already injects external spatial features into a ViT through cross-attention and then extracts the adapted ViT features using a second cross-attention module. BiSANet mainly applies this established interaction pattern between an RGB-T branch and a frozen SAM encoder.

  • The remaining components are also largely combinations of familiar designs. HDE combines difference features, convolutional fusion, spatial gating, and adaptive weighting, while SED uses SegFormer-style projections, ASPP, and auxiliary foreground and boundary supervision. Recent RGB-T methods have also adapted vision foundation models or SAM to this task, including HAPNet, SHIFNet, and SARTM. The paper therefore needs to explain more clearly what is conceptually new beyond applying an existing bidirectional adapter pattern to RGB-T and SAM features.

[1] HAPNet: Toward Superior RGB-Thermal Scene Parsing via Hybrid, Asymmetric, and Progressive Heterogeneous Feature Fusion (Li et al., Biomimetic Intelligence and Robotics, 2026)

[2] SHIFNet: Unveiling the Potential of Segment Anything Model 2 for RGB-Thermal Semantic Segmentation with Language Guidance (Zhao et al., IROS 2025)

[3] SARTM: Segment Any RGB-Thermal Model with Language-aided Distillation (Xing et al., arXiv 25).

W2. Incomplete Recent Comparisons and Unsupported State-of-the-Art Claim

  • Several directly relevant results on the same datasets are omitted. Complementary Random Masking for RGB-Thermal Semantic Segmentation (Shin et al., ICRA 2024) reports 61.2% mIoU on MFNet, tying the submitted result. Frequency-Guided Fusion for RGB-Thermal Semantic Segmentation (Canitez and Erkent, CVPR Workshops 2026) reports 61.28% without test-time augmentation and 61.73% with test-time augmentation on MFNet, using only 35.43M parameters and 51.39 GFLOPs. On PST900, SHIFNet reports 89.8% mIoU and SARTM reports 89.88%, both exceeding BiSANet's 88.2%. SHIFNet also reports 67.8% on FMB, only 0.1 percentage points below BiSANet's 67.9%. Consequently, the abstract and experimental discussion do not support the claim that BiSANet achieves state-of-the-art performance on all three benchmarks.

W3. The Semantic-Alignment Evidence Is Not Causal

  • The increased CKA and reduced MMD do not by themselves demonstrate that the model learns more useful target-domain semantics. The injector and extractor explicitly mix the two feature streams through cross-attention, so increased representation similarity is partly expected by construction. The paper does not show that the alignment metrics correlate with class-wise segmentation accuracy, robustness, or semantic correctness. It also does not compare against simpler bidirectional fusion mechanisms that may produce similar feature similarity.
Questions for authors

Q1. Novelty relative to ViT-Adapter

  • How does the proposed Bidirectional Semantic Adapter differ technically from the Spatial Feature Injector and Multi-Scale Feature Extractor of ViT-Adapter (Chen et al., ICLR 2023), beyond changing the interacting feature streams to RGB-T and SAM features? Please provide a module-level comparison and results from the closest ViT-Adapter-style baseline.

Q2. Missing recent methods and the state-of-the-art claim

  • Can the authors add results or a protocol-matched discussion for Complementary Random Masking, HAPNet, SHIFNet, SARTM, and Frequency-Guided Fusion? Given the results reported by these methods, on which benchmark and evaluation protocol is BiSANet actually state of the art?

Q3. Evidence for semantic alignment

  • Can the authors demonstrate that CKA or MMD improvement predicts segmentation improvement across adapter variants, training seeds, classes, or illumination conditions? How does the full adapter compare with simpler two-way cross-attention, residual fusion, or a parameter-matched ViT-Adapter interaction block?
Reviewer_kGjCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BiSANet for RGB-thermal (RGB-T) semantic segmentation, addressing the limited interaction between RGB-pretrained foundation-model priors and multimodal features in existing methods. It comprises a Hierarchical Dual-Modal Encoder (HDE) that fuses RGB and thermal features, Bidirectional Semantic Adapters (BSA) that enable information exchange between a frozen SAM ViT-L and RGB-T representations, and a Semantic Enhancement Decoder (SED) with auxiliary foreground/background and edge prediction heads. BiSANet reports mIoU scores of 61.2% on MFNet, 88.2% on PST900, and 67.9% on FMB. The paper also evaluates semantic alignment using CKA, MMD, and t-SNE, and presents ablations on model components, loss functions, adapter positions, day/night conditions, and computational efficiency.

Critical strengths
  1. Extensive experiments. The method is evaluated on three RGB-T benchmarks and compared with many existing methods. The paper includes ablations, adapter comparisons, day/night results, efficiency analysis, and failure cases. It also reports results using lighter SAM ViT-B and MobileSAM variants.

  2. Relevant problem with some supporting evidence. Using RGB-pretrained foundation-model priors for RGB-T segmentation is a worthwhile direction. At night, the full adapter outperforms the extractor-only variant by 3.0 mIoU (Table 9), suggesting that feeding RGB-T cues back into the SAM pathway may help when RGB appearance becomes less reliable.

Critical weaknesses
  1. Limited novelty. The BSA closely resembles ViT-Adapter (ICLR 2023), using cross-attention to inject features into ViT tokens and extract them back, with a zero-initialized scale. ViT-Adapter is neither cited nor compared, making the main contribution appear incremental.

  2. Combination of existing components. The difference-based gating, softmax branch weighting, and SegFormer-style decoder with ASPP and auxiliary supervision are established techniques. The paper offers limited methodological novelty beyond combining them.

  3. Gains may reflect model capacity. BiSANet uses 419.8M parameters and 556 GFLOPs, substantially more than most baselines, yet improves over TUNI by only 0.6–1.6 mIoU across the three datasets.

  4. No clear advantage with smaller models. The SAM ViT-B and MobileSAM variants fail to outperform the strongest baselines on MFNet. TUNI is also missing from the efficiency comparison, making it difficult to assess performance at comparable computational costs.

  5. Important baselines are missing. IG-SAM, CMNeXt, DFormer/DFormerv2, and Sigma should be included to provide a more complete comparison with related SAM-based and RGB-X methods.

  6. Simpler SAM alternatives are not evaluated. The paper lacks comparisons against feeding thermal images directly into SAM, fine-tuning SAM with LoRA or adapters, and concatenating SAM features into the decoder. It is therefore unclear whether bidirectional adaptation is necessary.

  7. Ablation results are small and lack variance estimates. Most component improvements are only 0.3–1.0 mIoU on MFNet, which has 393 test images. Without repeated runs or uncertainty estimates, it is difficult to determine whether these gains are reliable. Moreover, the extractor-only variant performs slightly worse than having no adapter.

  8. The benefits appear concentrated at night. BiSANet underperforms TUNI during daytime, and the full adapter performs worse than the extractor-only variant. This limits the evidence that bidirectional adaptation provides consistent benefits across conditions.

Questions for authors
  1. Novelty and alternative designs. How does BSA differ from ViT-Adapter’s injector/extractor design? Please compare it directly with a ViT-Adapter-style interaction and simpler SAM alternatives, including thermal input to SAM, LoRA/adapter fine-tuning, and concatenating SAM features into the decoder.

  2. Fair comparison with existing methods. Can you compare BiSANet with IG-SAM, CMNeXt, DFormerv2, and Sigma under comparable parameter and FLOP budgets, ideally using ViT-B-scale encoders? Please also include TUNI in Table 11.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.