- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BiSANet for RGB‑T semantic segmentation, which builds bidirectional semantic alignment between RGB‑T multimodal features and frozen SAM pre‑trained semantic priors. The model consists of three core modules: Hierarchical Dual‑Modal Encoder (HDE), Bidirectional Semantic Adapter (BSA), and Semantic Enhancement Decoder (SED). HDE implements level‑wise cross‑modal fusion for RGB‑T inputs. BSA contains Injector and Extractor components: Injector injects RGB‑T target‑domain information into SAM transformer blocks to adapt pre‑trained semantics, and Extractor transfers adapted SAM priors back to calibrate multimodal representations. SED fuses multi‑level features together with auxiliary loss heads for foreground‑background and boundary supervision.
Critical strengths
- The research motivation is clear: existing foundation‑model‑driven RGB‑T segmentation mainly adopts unidirectional prior injection, ignoring domain adaptation of pre‑trained RGB semantic priors toward thermal modality. The bidirectional interaction perspective is reasonable.
- Comprehensive benchmark evaluations are performed on three mainstream RGB‑T semantic segmentation datasets, with rich ablation studies covering module components, adapter variants, loss combinations, and encoder efficiency trade‑offs. Visualization analyses including t‑SNE, CKA, MMD, weight visualization and residual response maps are provided to interpret model behaviours.
- The hierarchical fusion design inside HDE considers different characteristics of low‑level spatial details and high‑level semantic features for RGB‑T data. Auxiliary supervision for boundary and foreground‑background further improves segmentation boundary quality.
Critical weaknesses
- Limited core novelty. The bidirectional adapter is built upon vanilla cross‑attention. The injector and Extractor are essentially two independent cross‑attention layers without special architectural innovation. Similar bidirectional cross‑modal interaction patterns have been widely explored in prior RGB‑T segmentation and multimodal foundation‑model adaptation works. The main gain partially comes from the heavily frozen SAM ViT‑L backbone rather than purely from the proposed bidirectional alignment mechanism.
- Insufficient baseline comparison for foundation‑model setting. The paper lacks fair baselines that adopt the same frozen SAM backbone with only unidirectional adapter modifications. The current baseline uses simple MFNet‑style fusion, which is not a strong representative of existing SAM‑based RGB‑T methods. It cannot solidly prove that performance improvement originates from bidirectional design instead of stronger fusion or decoder.
- Ambiguity on gain attribution. Table 5 ablation shows that standalone BSA brings marginal improvement only (+0.5 mIoU). The bidirectional adapter achieves large performance boost only when combined with HDE. It is hard to tell how much improvement is genuinely contributed by bidirectional semantic alignment itself. Furthermore, the nighttime gain shown in Table 9 is promising, but the paper lacks a quantitative statistical significance test to verify whether such improvement is stable and non‑random.
- Efficiency and practical concerns. When equipped with SAM ViT‑L, the model only runs at 11.4 FPS with 556.4 G FLOPs, which severely limits real‑world deployment for autonomous driving and surveillance scenarios targeted by RGB‑T segmentation. Although ViT‑B / ViT‑T lightweight variants are provided, noticeable performance drop exists. In addition, the paper does not conduct inference‑time complexity analysis for the proposed adapter modules.
- Missing critical analysis on failure cases. The failure case discussion is superficial. The authors only mention extremely small‑object and low‑contrast failure modes, yet do not provide actionable insight for how bidirectional alignment fails under these challenging conditions. There is no discussion about failure difference between unidirectional versus bidirectional adapters on hard samples.
- Writing drawback. Some descriptions are vague. For instance, the design motivation for adapter insertion positions (11,17,23 blocks of SAM) lacks sufficient theoretical justification, only supported by ablation results.
Questions for authors
- In Table 5, standalone BSA only brings marginal mIoU improvement (+0.5%). The large performance gain emerges only when BSA is combined with HDE. Could you quantitatively decompose and report the isolated performance contribution purely from your bidirectional alignment mechanism (Injector+Extractor), under a fixed, strong multimodal fusion baseline that is already equipped with frozen SAM? The current baseline adopts simple MFNet‑style fusion, which cannot rule out that gains come from improved fusion rather than bidirectional cross‑attention.
- Your key claim is that bidirectional adaptation yields large improvements especially under night / low‑light conditions (Table 9). Are the nighttime mIoU improvements of +3.0% (vs Extractor‑only) statistically significant? Please provide paired statistical test results (e.g., paired t‑test across test splits / multiple repeated runs with different random seeds) to exclude the possibility that this observed gain arises from random training fluctuation.
- Why are SAM blocks 11, 17, 23 selected as insertion positions for Bidirectional Semantic Adapters besides ablation results? Is there any architectural or theoretical motivation, other than brute‑force ablation over candidate transformer indices?
- What is the exact additiona/ latency) introduced only by the Injector‑Extractor adapter modules, decoupled from the heavy SAM ViT‑L backbone? It is difficult to judge whether the adapter itself introduces substantial inference cost.
- In failure‑case analysis, for those hard samples with tiny, low‑contrast objects: quantitatively compare prediction errors (mIoU on failure subset) between unidirectional variants (Injector‑only / Extractor‑only) and full bidirectional BSA. Does bidirectional alignment still bring a consistent advantage on these challenging samples, or does it fail equally as unidirectional prior injection?