acceptodds
Withdrawn from ICLR 2027

Rethinking Unaligned RGB-T Dense Prediction as Cross-Modal Information Transport

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

RGB-T dense prediction remains challenging because visible and thermal observations are rarely perfectly aligned in practice. When the two modalities are spatially inconsistent, the thermal response at a given coordinate may originate from a different scene region, and reweighting such co-located responses can only attenuate the mismatch rather than recover the complementary information that lies elsewhere. We therefore propose unaligned cross-modal information transport (UCMIT), which reformulates RGB-T interaction as a structured cross-modal information transport process. Within UCMIT, local cross-modal information transport (LCIT) constructs a semantic-geometric transport plan through entropy-regularized optimal transport, retrieving complementary thermal evidence from plausible nearby locations and reconstructing it in the visible coordinate system without explicit global registration. Source-side marginal constraints couple evidence allocation across RGB targets and prevent unrestricted reuse of dominant thermal responses, while a fixed window-level null mass is redistributed across targets according to their relative matching preference. Reliability-controlled evidence fusion (REF) then arbitrates between the co-located and transported responses according to the local transport state and regulates residual injection according to evidence reliability and path-aware certainty. LCIT thus determines where complementary evidence is obtained, while REF determines which evidence is retained and how strongly it modifies the visible representation. Extensive experiments on RGB-T salient object detection and semantic segmentation demonstrate strong overall performance across unaligned benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RGB-T Dense Prediction, Unaligned Multimodal Fusion, Cross-Modal Information Transport, Salient Object Detection, Semantic Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Td6VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the cross‑modal misalignment problem for RGB‑T dense prediction tasks including salient object detection and semantic segmentation. The authors propose UCMIT, a two‑stage pipeline consisting of Local Cross‑Modal Information Transport (LCIT) and Reliability‑Controlled Evidence Fusion (REF). LCIT builds semantic‑geometric optimal transport plans within local windows to reconstruct RGB‑aligned thermal evidence without explicit image‑level registration. REF arbitrates between co‑located thermal features and transported thermal evidence, and modifies RGB representations via reliability‑aware residual injection.

Critical strengths
  1. The problem setting is practically meaningful. Real‑world RGB‑T sensors commonly suffer from parallax and calibration errors, so mitigating unaligned multimodal fusion is a valuable research direction.
  2. The technical idea of separating evidence retrieval (transport) and representation update (fusion) is conceptually clear. Using entropy‑regularized optimal transport with null‑source marginal constraints to avoid over‑reusing dominant thermal responses is an interesting design choice.
  3. Comprehensive evaluations are provided: the paper validates UCMIT on two different dense prediction tasks, covers quantitative comparison, qualitative visualization, ablation studies, robustness tests under spatial shift and thermal noise, and efficiency analysis. Supplementary materials further include hyper‑parameter sensitivity, multi‑scale feature visualization and failure‑case discussion.
  4. The implementation details are well‑documented, and anonymous source code is promised for reproducibility.
Critical weaknesses
  1. Limited technical novelty of optimal‑transport formulation. The adoption of Sinkhorn‑based entropy‑regularized optimal transport for visual correspondence is well‑studied in existing literature. The source‑side marginal constraint and null‑source design are incremental modifications rather than fundamental methodological innovation. The insight of decoupling matching and fusion is partially seen in prior alignment‑free multimodal works.
  2. Critical practical limitation of local‑window transport. The whole LCIT operates within fixed‑size local spatial windows. As acknowledged by the authors in the failure‑case analysis, once actual cross‑modal displacement exceeds the predefined window range, correct thermal evidence cannot be retrieved. The paper lacks an effective solution for large, spatially varying misalignment; simply tuning window size brings heavy computational overhead. This greatly restricts real‑world applicability.
  3. Insufficient baseline comparisons against state‑of‑the‑art alignment‑free modules. The paper compares final task performance, but lacks direct module‑level controlled comparison against deformable convolution, correlation‑based alignment, and recent correspondence learning modules under exactly identical backbone. Some strong recent alignment‑free RGB‑T works are only briefly cited without detailed module‑wise analysis.
  4. Concerns about complexity and practical deployment. Even with local‑window restriction, repeated Sinkhorn iterations across multi‑scale feature maps bring non‑negligible computational overhead. Although FPS is reported, memory consumption is not reported. It is unclear whether this OT‑based pipeline can be deployed on edge platforms typical for RGB‑T perception.
  5. Ambiguity about hyper‑parameter robustness. While hyper‑parameter sensitivity plots are given, many key hyper‑parameters (window size, null‑source mass ρ, geometric weight λ_g, Sinkhorn iteration number) are manually tuned. It remains unclear how well these hyper‑parameters transfer to new, unseen datasets with different misalignment distributions.
  6. Weak insight from theoretical analysis. The appendix gives a mathematical derivation for source‑coupled softmax, yet there is no formal analysis of convergence, error bounds, or how transport hyper‑parameters interact with misalignment magnitude. The analysis remains empirical.
Questions for authors
  1. Current comparisons mainly report end‑to‑end task metrics. To isolate the effectiveness of your transport mechanism itself: Could you provide module‑level controlled comparisons: plug your LCIT module, deformable sampling, and correlation‑based alignment into an identical decoder/loss/backbone, on the same validation split, and report intermediate feature‑level metrics (e.g., feature alignment error between the transported thermal feature and the ground‑truth aligned thermal feature), besides the final S‑measure / mIoU?
  2. The core LCIT module operates within fixed‑size local windows. In failure cases, the paper shows performance drops when misalignment exceeds window boundaries. Could you clarify: (1) What is the statistical distribution of real misalignment magnitude in your test benchmarks (UVT20K, KUST4K, U‑MFNet)? What percentage of real‑world samples contain thermal‑RGB displacement larger than your adopted local window size? (2) Have you quantified performance degradation when misalignment gradually grows beyond the window radius? Could you provide controlled quantitative curves for this?
  3. Your row‑wise softmax ablation demonstrates that source‑side marginal coupling brings gains. However, the source marginal budget is manually fixed. (1) Why is it preferable to fix the source marginal budget instead of learning it data‑dependently? Are there theoretical or empirical risks if we learn this marginal dynamically? (2) When transferring to a new dataset with drastically different misalignment statistics, how do you tune ρ and λ_g without exhaustive grid search?
Reviewer_jtGoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies dense prediction from spatially unaligned RGB and thermal images. Its main observation is that thermal information at the same pixel coordinate may come from a different scene region, so simply reweighting co-located features cannot recover the correct complementary evidence. The proposed UCMIT framework uses Local Cross-modal Information Transport (LCIT) to construct a local optimal-transport plan from semantic and geometric costs and reconstruct thermal evidence in RGB coordinates. A null source handles locally implausible matches, while Reliability-controlled Evidence Fusion (REF) chooses between the original co-located feature and the transported feature before gating the residual update to the RGB stream. Separate models are trained for RGB-T salient object detection and semantic segmentation. Although the method obtains competitive results, the current experiments do not sufficiently support the broad claims of generality and robustness to real-world misalignment.

Critical strengths

None

Critical weaknesses

W1. Limited Novelty Relative to Existing Unaligned RGB-T Methods

  • The overall idea substantially overlaps with prior unaligned RGB-T methods. Existing approaches already perform affine or deformable alignment, cross-modal correspondence estimation, RGB-conditioned thermal feature sampling, and alignment-before-fusion [1–6]. UMFNet also uses uncertainty to suppress unreliable cross-modal information [7]. Therefore, retrieving non-co-located thermal evidence and controlling its reliability are not themselves new.

[1] “Weakly Alignment-Free RGBT Salient Object Detection With Deep Correlation Network,” TIP 2022.

[2] “Alignment-Free RGBT Salient Object Detection: Semantics-Guided Asymmetric Correlation Network and a Unified Benchmark,” T-MM 2024.

[3] “Alignment-Free RGB-T Salient Object Detection: A Large-Scale Dataset and Progressive Correlation Network,” AAAI 2025.

[4] “Breaking Alignment Barriers: TPS-Driven Semantic Correlation Learning for Alignment-Free RGB-T Salient Object Detection,” AAAI 2026.

[5] “Deformation-Resilient Multigranularity Learning for Unaligned RGB-T Semantic Segmentation,” TNNLS 2025.

[6] “Implicit Alignment and Query Refinement for RGB-T Semantic Segmentation,” Pattern Recognition, 2026.

[7] “Uncertainty-Aware Modality Fusion for Unaligned RGB-T Salient Object Detection,” CVPR, 2026.

W2. Unfair Comparison Setting

  • UCMIT is jointly trained on the official training splits of UVT20K and un-VT5000. In contrast, the standard alignment-free protocol introduced with PCNet trains the compared models only on UVT20K and then evaluates them on UVT20K, UVT2000, and the three un-VT datasets. Alignment-Free RGB-T Salient Object Detection: A Large-Scale Dataset and Progressive Correlation Network (Wang et al., AAAI, 2025) explicitly describes this UVT20K-only protocol.

  • Consequently, UCMIT receives additional labeled examples from the un-VT5000 training distribution, while most comparison methods copied into Table 1 were obtained without this data. This is particularly problematic for un-VT5000 and may also help on un-VT1000 and un-VT821 because all three are produced using the same synthetic affine-deformation procedure. The authors should report a UVT20K-only UCMIT model and either retrain the strongest baselines with the joint dataset or clearly separate the two protocols.

W3. Claim-Experiment Mismatch: KUST4K Is Not an Unaligned Benchmark

  • The paper repeatedly describes KUST4K as unaligned and uses it as the main semantic-segmentation evaluation for spatial inconsistency. However, the official KUST4K paper describes the dataset as containing pixel-aligned RGB-TIR pairs. It first applies SIFT-based registration and then uses manual registration to ensure accurate spatial alignment. Therefore, good performance on KUST4K supports RGB-T segmentation accuracy, but it does not validate robustness to naturally unaligned inputs.

  • A directly relevant missing evaluation is URTF, introduced in Graph-based Semantic Calibration Network for Unaligned UAV RGBT Image Semantic Segmentation and a Large-scale Benchmark (Fan et al., arXiv, 2026). URTF contains more than 25,000 RGB-T pairs, 61 classes, and realistic spatial offsets caused by sensor parallax and UAV vibration.

W4. Missing Recent Baselines and Incomplete SOTA Positioning

  • The segmentation comparison omits several recent methods. On the standard MFNet benchmark, UCMIT reports 57.8% mIoU, whereas: 1) Seeing Roads Through Words: A Language-Guided Framework for RGB-T Driving Scene Segmentation (Reddy et al., ICIP, 2026) reports 62.3%, 2) Frequency-Guided Fusion for RGB-Thermal Semantic Segmentation (Canıtez and Erkent, CVPR Workshops, 2026) reports 61.28% without test-time augmentation and 61.73% with it, 3) Segment Any RGB-Thermal Model with Language-aided Distillation (Xing et al., arXiv, 2025; revised 2026) reports 60.03%.

  • The SOD comparison is also incomplete. Unaligned Red–Green–Blue and Thermal Salient Object Detection via Structure-Aware Prior Guidance (Gao et al., Engineering Applications of Artificial Intelligence, 2026), referred to as SPGNet, is cited in the paper but not quantitatively compared. Geometry-Aware Vision Mamba for Dual-Modal Salient Object Detection (Gao et al., IJCAI, 2026) is also relevant, but omitted from the paper. Their omission means that UCMIT cannot be presented as broadly superior in RGB-T SOD/semantic segmentation tasks.

W5. Limited Benchmark Diversity

  • The SOD evaluation appears broad because it contains five datasets, but three are synthetic weakly aligned versions of the same established aligned datasets. Moreover, un-VT5000 contributes labeled training data to UCMIT. The evaluation does not include the aligned VT5000, VT1000, and VT821 benchmarks to determine whether transport harms performance when correspondence is already accurate.

  • The segmentation coverage is substantially weaker: one registered UAV dataset in the main paper, one aligned driving dataset (i.e.,MFNet dataset) in the appendix, and one synthetic deformation of that same driving dataset. Common RGB-T segmentation benchmarks covering different environments are absent, including PST900: RGB-Thermal Calibration, Dataset and Segmentation Network (Shivakumar et al., ICRA, 2020) and FMB from Multi-interactive Feature Learning and a Full-time Multi-modality Benchmark for Image Fusion and Segmentation (Liu et al., ICCV, 2023).

W5. No Evaluation Across Misalignment Severity or Type

  • The paper does not evaluate performance as a function of misalignment magnitude. Table 4 applies random thermal translation within a single ±50-pixel range and reports only the average drop. U-MFNet uses only the prescribed deformation strength of 20 pixels. Natural misalignment in UVT20K and UVT2000 is not measured or divided into mild, moderate, and severe subsets.

  • The robustness study should include displacement curves and multiple transformation families, such as 1) Translation at several magnitudes, 2) Rotation and scale changes, 3) Affine and local non-rigid deformation, 4) Sensor parallax and partial field-of-view overlap, 5) Occlusion, modality dropout, and combined geometric and thermal corruption.

  • This analysis is particularly important because the paper acknowledges that LCIT fails when displacement exceeds its local window. At present, the experiments do not identify where that failure begins or how quickly performance degrades.

Questions for authors

Q1. Matched SOD Training Protocol

  • What are UCMIT’s results when trained only on UVT20K, following Alignment-Free RGB-T Salient Object Detection: A Large-Scale Dataset and Progressive Correlation Network (Wang et al., AAAI, 2025)? Alternatively, can the authors retrain PCNet, TPS-SCL, and UMFNet using the same joint UVT20K plus un-VT5000 data?

Q2. KUST4K Alignment Status

  • Why is KUST4K described as unaligned when An RGB-TIR Dataset from UAV Platform for Robust Urban Traffic Scenes Semantic Segmentation (Ouyang et al., Scientific Data, 2025) explicitly applies SIFT and manual registration? Can the authors evaluate on the real unaligned URTF benchmark introduced by Fan et al. (arXiv, 2026)?

Q3. Robustness Across Misalignment Severity

  • Can the authors report performance curves over multiple translation, rotation, scale, affine, and local-warp strengths? For the naturally unaligned UVT20K and UVT2000 datasets, can samples be divided according to an estimated displacement or registration-error measure?

Q4. Recent SOD/Segmentation Comparisons

  • Why are recent models such as Frequency-Guided Fusion (Canıtez and Erkent, CVPR Workshops, 2026), CLARITY (Reddy et al., ICIP, 2026), HAPNet (Li et al., Biomimetic Intelligence and Robotics, 2026), SHIFNet (Zhao et al., IROS, 2025), and SARTM (Xing et al., arXiv, 2025/2026) omitted? Can the authors evaluate UCMIT on PST900 and FMB and update the MFNet comparison?

  • How does UCMIT compare under a matched protocol with SPGNet (Gao et al., Engineering Applications of Artificial Intelligence, 2026) and Geometry-Aware Vision Mamba for Dual-Modal Salient Object Detection (Gao et al., IJCAI, 2026)?

Reviewer_7X4DOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses dense prediction from unaligned RGB–thermal image pairs and proposes UCMIT, a framework that reformulates RGB–thermal interaction as structured cross-modal information transport. The Local Cross-Modal Information Transport (LCIT) module combines semantic compatibility with geometric proximity within local windows to establish correspondences through entropy-regularized optimal transport, subject to source-side marginal constraints and a fixed window-level null-source budget. The resulting transport plan reconstructs thermal evidence in the RGB coordinate system. The Reliability-Controlled Evidence Fusion (REF) module then adaptively balances co-located thermal responses against transported evidence and regulates their residual contribution to the RGB representation based on evidence reliability and path-aware certainty. By separating evidence retrieval and reconstruction from representation updates, the framework allows thermal evidence to be assessed before it influences the RGB stream, reducing the impact of spatially mismatched or unreliable responses while retaining useful complementary information. Experiments on RGB–thermal salient object detection and semantic segmentation demonstrate strong overall performance across the evaluated benchmarks.

Critical strengths

Strengths:

  1. The paper identifies an important limitation of position-wise fusion: when a co-located thermal feature originates from an incorrect scene region, reweighting it can attenuate the resulting interference but cannot recover complementary information located elsewhere. This observation provides a convincing motivation for neighborhood-based thermal evidence retrieval. The proposed separation between evidence reconstruction and representation update is also well justified.

  2. LCIT reconstructs thermal evidence before REF selects the evidence and controls its contribution to the RGB representation. Preserving the direct path accommodates valid co-located correspondences, while the shared arbitration weight keeps path-aware certainty consistent with evidence selection.

  3. The method achieves competitive or leading results across multiple benchmarks for both salient object detection and semantic segmentation.

Critical weaknesses

Weaknesses:

  1. Table 5 compares fusion variants but lacks comparisons with representative complete models from Tables 1 and 2. Reporting their parameters, FLOPs, and inference speed under comparable conditions would clarify the trade-off between accuracy and efficiency.

  2. Section 4.2 refers to “unaligned KUST4K,” whereas the original dataset paper describes registered, pixel-aligned pairs. Please clarify whether additional perturbations or substantial residual misalignment are involved, and whether all methods follow the same protocol.

  3. Please briefly describe SαS_\alpha, EξE_\xi, and FβwF_\beta^w, cite their original formulations, and specify the metric variants, relevant parameters, prediction post-processing, and evaluation toolkit.

  4. Unclear LCIT window and boundary handling. Please specify whether windows overlap or are shifted, how padded tokens affect matching and transport marginals, and whether correspondences across window boundaries are accessible. These details determine the effective local search capability.

  5. The experimental conditions of the SOTA comparisons require clarification. Although Table 3 uses a controlled setup and UCMIT’s training details are provided, the conditions underlying Tables 1 and 2 remain unclear. For each compared method, please specify whether the metrics were taken from prior publications or reproduced by the authors, and report relevant differences in training and evaluation settings.

  6. Please correct repeated equation-reference errors, such as “Eq. equation 1” and “Eq. equation 2.”

Questions for authors

Please address the clarification questions raised in the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.