acceptodds
Withdrawn from ICLR 2027

Calibrate What You Trust: Reliable Multimodal 3D Panoptic Segmentation under Domain Shift

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present the first study of Domain Generalization (DG) for multimodal 3D panoptic segmentation (mm-3DPS), targeting robust perception under unseen domain shifts in autonomous driving. We identify two key challenges: reliable multimodal representation learning when individual modalities are affected differently by domain shifts, and reliable panoptic prediction when these shifts propagate to the final scene decomposition. To address them, we propose RAMP3D, a DG framework specifically designed for mm-3DPS. RAMP3D incorporates Reliability-guided Multimodal Adaptation (ReMA), which explicitly estimates modality reliability and adaptively regulates multimodal adaptation under domain shift. It further introduces Reference-based Calibration (ReCal), which leverages stable scene-relative cues to separately calibrate stuff-region predictions and thing-instance scores. Extensive experiments across diverse shifts in time, weather, location, and sensor conditions show that RAMP3D consistently outperforms strong baselines, including adaptations of state-of-the-art 3D DG and multimodal UDA methods.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, 3D Panoptic Segmentation, Domain Generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RWQzOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates domain generalization for multimodal 3D panoptic segmentation and proposes RAMP3D, a framework designed to improve robustness under unseen domain shifts. The proposed framework consists of two main components: Reliability-guided Multimodal Adaptation (ReMA) and Reference-based Calibration (ReCal). ReMA estimates modality-specific reliability through shared learnable tokens and cross-modal evidence, dynamically adjusting image- and LiDAR-conditioned feature updates. ReCal introduces road-relative representations to refine stuff-region predictions and recalibrate thing-instance confidence scores. This work evaluates RAMP3D on nuScenes, SemanticKITTI, and Waymo under illumination, weather, geographic, and sensor-related domain shifts. The reported results demonstrate improvements over several adapted domain generalization and multimodal segmentation baselines.

Critical strengths
  1. The paper investigates multimodal 3D panoptic segmentation under domain generalization, which is a practically relevant.

  2. The proposed framework considers robustness at both the feature-representation and panoptic-prediction levels, providing a coherent overall design.

  3. The experiments cover multiple domain shifts and report substantial improvements over several baseline methods.

Critical weaknesses
  1. This work is not the first to investigate Domain Generalization for 3D panoptic segmentation. Frontier works include:

[1] PanDA: Unsupervised Domain Adaptation for Multimodal 3D Panoptic Segmentation in Autonomous Driving

  1. Only PQ is reported as the evaluation metric in this work. It would be valuable to see comparisons of mIoU, PQstPQ^{st}, and PQthPQ^{th}.

  2. The comparison in Table 1 is not fair. UniDSeg adopts a CLIP image encoder, while this work uses a DINOv2 image encoder.

  3. This work does not report results for two settings: training and evaluation on the target domain, and training on combined source and target data. It remains unclear whether substantial gaps persist compared with these methods.

Questions for authors

Please refer to the above weaknesses and further answer questions.

Q1: Could results be supplied for two experimental setups: training and evaluation on the target domain, and training on combined source and target data? Do substantial performance gaps still exist between the proposed approach and competing methods under these settings?

Q2: Only PQ is reported as the evaluation metric in this work. Could comparisons of mIoU, PQstPQ^{st}, and PQthPQ^{th} be provided?

Q3: This work is not the first to investigate Domain Generalization for 3D panoptic segmentation. Could the discussion elaborate on the differences between this method and prior frontier work such as PanDA?

5 comments
AuthorsOpenReview ↗

Response to W1/Q3 & W2/Q2: DG–UDA Distinction and Additional Evaluation Metrics

W1 & Q3. Distinction from PanDA.

We respectfully disagree. PanDA studies Unsupervised Domain Adaptation (UDA), whereas RAMP3D addresses single-source Domain Generalization (DG). These are fundamentally different tasks: UDA requires unlabeled target-domain data for target-specific adaptation, while DG trains exclusively on source-domain data to generalize to multiple unseen target domains without retraining. We summarize these differences in Table R8 below. Notably, our manuscript already identifies PanDA as a UDA method in Sec. 2 (L116–117).

Table R8. Comparison of UDA and single-source DG settings.

MethodSettingSource training dataTarget training dataTarget-specific training
PanDAUDA✓✓ (unlabeled)✓
Ours (RAMP3D)single-source DG✓✗✗

PanDA and RAMP3D differ not only in setting but also in method design. PanDA employs a teacher–student framework, where an EMA-updated teacher generates and refines pseudo-labels to supervise learning on unlabeled target-domain data. This target-dependent training mechanism cannot be directly applied to the source-only DG setting.

In contrast, RAMP3D is trained exclusively on source-domain data and generalizes to multiple unseen targets without further adaptation. Specifically, it (i) estimates modality-specific reliability from cross-modal evidence learned solely on the source domain (Sec. 3.2), and (ii) calibrates panoptic predictions using within-frame road-relative features without requiring target-domain statistics (Sec. 3.3).

W2 & Q2. Additional Evaluation Metrics.

We thank the reviewer for the suggestion. We have added mIoU, PQ-th, and PQ-st to Tables 2 and 3, providing further insights into the effectiveness of ReMA and ReCal.

  1. Effectiveness of ReMA. Global and local gating exhibit complementary benefits. Compared with no gating, global gating mainly improves PQ-st (+1.32/+2.69 on nighttime/Singapore), whereas local gating mainly benefits PQth (+4.88 on nighttime). Combining both improves PQ-th and PQ-st across all three targets, with gains of 5.85/1.39/1.44 PQ-th and 0.16/0.92/2.76 PQ-st on nighttime/rainy/Singapore. These results support the design of global and local gating.

  2. Effectiveness of ReCal. Stuff calibration improves PQ-st by 2.52/0.65 on nighttime/Singapore, while thing re-scoring improves PQ-th by 1.35 on both nighttime and rainy. Combining both achieves the highest overall PQ across all three targets, with gains of 2.72/0.82/0.17, respectively, confirming their complementary benefits for panoptic prediction.

AuthorsOpenReview ↗

Response to W3–W4 & Q1: Fair Comparisons and Target-Supervised Evaluation

W3. Fair Comparison with UniDSeg.

CLIP-B is an officially evaluated visual backbone in the original UniDSeg paper, which we adopt when adapting UniDSeg to our panoptic framework. Importantly, our original submission already reports RAMP3D results using the same CLIP-B backbone in Appendix Table 7.

We further extend this matched-backbone comparison to cross-dataset generalization, summarizing both intra-nuScenes and cross-dataset results in Table R9. RAMP3D consistently outperforms UniDSeg under the same CLIP-B backbone, achieving gains of 4.28/2.67/7.33/10.58 PQ on nighttime/rainy/SemanticKITTI/Waymo. Moreover, these gains exceed the performance variations caused by switching RAMP3D's 2D backbone from CLIP-B to DINOv2-B, demonstrating that the improvements primarily stem from our method design rather than the choice of 2D backbone.

Table R9. Fair comparison with UniDSeg across visual backbones. Italicized columns indicate source-domain results.

Method2D BackboneUS S.D. PQNighttime PQRainy PQSingapore PQnuScenes PQsKITTI PQWaymo PQ
UniDSegCLIP-B79.0144.1473.2963.0980.4340.7532.27
RAMP3DCLIP-B80.2948.4275.9667.2882.2748.0842.85
RAMP3DDINO-v2-B80.1252.0677.6569.0282.1846.1343.80

W4 & Q1. Comparison with target-supervised oracles.

Table R10 compares models that share the same 2D and 3D backbones but are trained on source-only, target-only, or combined source and target data, each evaluated on the corresponding target domain. The target-supervised oracles serve as empirical upper-bound references for assessing the remaining gap between DG and target-supervised learning.

Table R10. Comparison with target-supervised oracle models.

MethodTraining SetNighttime PQRainy PQSingapore PQsKITTI PQWaymo PQ
Baseline (IAL+DINOv2)source42.0871.9964.6935.9026.39
RAMP3D (DG)source52.0677.6569.0246.1343.80
Oracle (target only)target62.0076.2479.6267.4163.05
Oracle (source + target)source+target64.6179.9580.40––

Despite having no access to target-domain data, RAMP3D substantially narrows the gap to the target-only oracle and even surpasses it on rainy (77.65 vs. 76.24). The rainy result is likely explained by training-set size: the rainy training set (5,582 frames) is about half the size of RAMP3D's source set (Boston sunny-daytime, 10,635 frames), so the target-only oracle is data-limited. Once trained on both sets, the oracle reaches 79.95 and exceeds RAMP3D. The gap between RAMP3D and the oracle is larger on SemanticKITTI and Waymo, reflecting the greater challenge of cross-dataset sensor and distribution shifts.

Reviewer_RWQzOpenReview ↗

Response to Authors

Thank you for conducting the additional experiments and providing the clarifications. Your responses have addressed my concerns.

Reviewer_5JPbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the performance degradation of multimodal 3D panoptic segmentation in a source-only domain generalization setting. To this end, the authors propose ReMA, which adjusts feature adaptation by estimating the reliability of RGB and LiDAR modalities, and ReCal, which corrects "stuff" masks and "thing" instance scores using road-relative features. Building upon the existing IAL framework, the authors evaluate performance regarding lighting, weather, and geographical shifts within the nuScenes dataset, as well as cross-dataset generalization performance on SemanticKITTI and Waymo.

Critical strengths
  • The authors established a systematic protocol for evaluating multimodal 3D panoptic segmentation under source-only domain generalization settings. The evaluation covers not only lighting, weather, and geographical shifts within nuScenes but also cross-dataset generalization to SemanticKITTI and Waymo. The method consistently outperforms existing approaches across diverse domain shifts, achieving state-of-the-art results: 52.06 PQ (Nighttime), 46.13 PQ (SemanticKITTI), and 43.80 PQ (Waymo).

  • The ablation study regarding key components is comprehensive. Using a fixed DINOv2 backbone and consistent training conditions, the authors separately verify the contributions of 2D/3D feature injection and reliability gating, and individually evaluate the thing re-scoring and stuff calibration within ReCal. Additionally, Table 7 in the appendix demonstrates that the method is not dependent on a specific backbone by showing results with DINOv2, CLIP, and various LiDAR backbones.

Critical weaknesses
  • The algorithmic novelty is limited. ReMA is a framework that extends the feature adaptation based on REIN's learnable tokens to a multimodal setting and incorporates gating mechanisms that leverage cross-modal feature discrepancies. Similarly, ReCal combines existing mask quality re-scoring with a calibration method that utilizes road-relative features. Consequently, these represent system-level contributions rather than novel algorithmic contributions within the individual methods themselves.

  • The core contribution of this method, reliability estimation, has not been directly validated. While Table 2 demonstrates performance improvements resulting from gating, it remains unverified whether the reliability weights predicted by the Router actually reflect the true reliability of the modalities. As the weight distributions across domains and the system's response to actual modality degradation are not shown, it is difficult to distinguish the specific effects of reliability estimation from those of general adaptive feature gating.

Questions for authors
  • Although Appendix Table 7 presents experiments using DINOv2 and CLIP backbones, results regarding cross-dataset generalization are missing. Please confirm whether the performance improvements observed on SemanticKITTI and Waymo are maintained with the CLIP-based RAMP3D model.

  • Please compare the distributions of RGB and LiDAR reliability weights predicted by the Router across different conditions: Daytime, Nighttime, Rainy, and Singapore. In particular, I would like to verify whether the RGB reliability weight actually decreases during Nighttime conditions.

  • Please analyze how the Router's reliability weights respond when RGB brightness or noise levels are altered, or when LiDAR point cloud density is incrementally reduced, for the same input.

  • Please verify whether using a road reference in ReCal offers advantages compared to using a global mean reference or no reference at all. Additionally, since the road reference relies on a predicted road mask, please examine the impact of road prediction errors on ReCal's performance.

5 comments
AuthorsOpenReview ↗

Response to W1–W2: Methodological Contributions and Reliability Evaluation

W1. Algorithmic Novelty.

We respectfully disagree that our contributions are merely system-level integrations. While feature adaptation and confidence re-scoring are not new concepts, ReMA and ReCal introduce problem-specific mechanisms that address challenges unique to domain generalization (DG) in multimodal 3D panoptic segmentation (mm-3DPS), which existing methods do not adequately address.

(1) ReMA vs. REIN. REIN adapts visual foundation models through learnable tokens but does not account for modality-specific degradation under unseen domain shifts, allowing unreliable features to negatively affect multimodal predictions. This limitation is empirically evident: under nighttime conditions, Table 2 shows that 2D, 3D, and joint feature injection without reliability gating (rows 3–5) all underperform the LiDAR-only baseline (46.98 PQ). ReMA goes beyond extending REIN to multimodal inputs by introducing region-wise, modality-specific reliability estimation and adaptive feature regulation, explicitly controlling how much each modality contributes to VFM adaptation. This design improves nighttime PQ to 49.34, demonstrating that straightforward multimodal feature injection is insufficient and that reliability-aware adaptation provides a meaningful algorithmic contribution.

(2) ReCal vs. conventional quality re-scoring. Existing mask-quality re-scoring methods primarily learn to predict mask quality from instance features, without explicitly addressing the confidence–quality mismatch induced by unseen domain shifts. Consequently, false-positive instances may receive overconfident scores under nighttime or rainy conditions. ReCal introduces scene-relative calibration using a frame-specific road reference, constructing road-relative descriptors to mitigate domain-dependent feature biases rather than relying solely on absolute instance features. Moreover, ReCal addresses both thing-instance confidence and stuff-region predictions, whose errors require different calibration strategies. Thus, its contribution extends beyond conventional mask-quality re-scoring to domain-aware calibration tailored to mm-3DPS.

W2. Validation of Reliability Estimation.

We thank the reviewer for the suggestion. We conduct two additional experiments: (1) cross-domain router weight distributions and (2) controlled modality perturbations to validate ReMA's reliability estimation. The first reveals modality-specific weight variations across target domains, while the second demonstrates that perturbing one modality selectively reduces its reliability weight, leaving the other modality largely unchanged. Together, these results support the effectiveness of ReMA's reliability-guided gating. Detailed results are provided in our responses to Q2–Q3 and Appendix Sec.F.1 & Sec.F.2 of the revised manuscript.

AuthorsOpenReview ↗

Response to Q1–Q4: Cross-Dataset Generalization and Reliability Analysis

Q1. Cross-dataset Generalization with CLIP-B.

We thank the reviewer for this suggestion. We further evaluate RAMP3D with DINOv2-B and CLIP-B on SemanticKITTI and Waymo, as shown in Table R6. Both variants substantially outperform the original IAL baseline and achieve comparable cross-dataset performance, demonstrating that RAMP3D maintains strong generalization across different visual foundation models.

Table R6. Cross-dataset generalization with different visual backbones. Italicized columns indicate source-domain results.

Method2D BackbonenuScenes PQsKITTI PQWaymo PQ
IAL (Baseline)SwiftNet79.6325.8520.15
RAMP3DDINOv2-B82.1846.1343.80
RAMP3DCLIP-B82.2748.0842.85

Q2. Reliability Distributions Across Domains.

In Appendix Sec. F1 and Fig.6, we visualize the distributions of RGB and LiDAR router weights across daytime, nighttime, rainy, and Singapore, measured at the first ReMA injection layer to avoid the influence of preceding feature adaptations. Since the two modalities independently modulate different residual branches, their absolute weight magnitudes are not directly comparable.

Comparing different domains within the same modality, we observe that RGB weights decrease under nighttime conditions, consistent with degraded visual cues. LiDAR weights also vary across domains, potentially reflecting differences in scene composition and sensing conditions. We therefore further investigate modality-specific responses using controlled perturbations on identical scenes.

Q3. Responses to Controlled Modality Degradation.

We independently perturb RGB brightness, introduce Gaussian noise, and reduce LiDAR point density while keeping the other modality unchanged. As shown in Fig. 6(b) and Appendix Sec. F.2, the router selectively reduces the reliability weight of the degraded modality, while leaving the other modality's weight largely unchanged. These results demonstrate that ReMA learns modality-specific reliability rather than indiscriminately adjusting both modalities.

Q4. Effectiveness and Robustness of the Road Reference.

(1) Road reference ablation. We replace road-local mean centering with global-mean centering or zero the appearance input to ReCal. As shown in Table R8, these modifications reduce average PQ by 0.42 and 2.26 points across the three target domains, respectively, supporting the effectiveness of scene-relative appearance features for panoptic calibration.

Table R7. PQ changes relative to road-local feature centering. Negative values indicate degradation.

Appearance inputSourceSingaporeNightRainy
Global-mean centering−0.01−0.33−0.81−0.13
Zero appearance input−0.93−2.12−3.51−1.15

(2) Robustness to road-mask errors. We further evaluate seven road-mask configurations: random road-voxel removal, distant road-voxel removal, neighboring-class leakage (add voxels predicted as neighboring classes), boundary dilation (expand road into neighboring classes), boundary erosion (remove road voxels near neighboring classes), ground-truth road masks, and empty masks. The results are presented in Fig. 7 and Appendix Sec. F.3.

ReCal tolerates substantial missing road coverage, while contamination from neighboring classes has a larger impact, particularly under nighttime conditions. However, the predicted road masks already achieve high IoUs (97.75/92.50/86.84/93.29% on four domains, marked in dashed lines), and replacing them with ground truth yields only marginal gains, indicating that road-mask errors have limited impact under our evaluated conditions.

Thank you for the additional experiments. Most of my concerns have been addressed, but I still have two questions.

(1) The authors explain the differences between ReMA and REIN. However, using a router to adjust modality contributions based on reliability is already a common approach. I still confuse the contribution of this module is mainly an effective combination of existing methods.

(2) In Figure 6(a) of Appendix, the weight distributions are quite similar, especially between sunny and rainy conditions. The changes in Figure 6(b) also appear small. Would artificially darkening sunny images produce a distribution similar to nighttime? More explanation is needed to show that these weights actually reflect modality reliability.

Reviewer_WZ6EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies single-source domain generalization for LiDAR-camera 3D panoptic segmentation. RAMP3D adapts a frozen DINOv2 with learnable tokens whose image- and LiDAR-conditioned residual updates are gated by estimated per-patch modality reliability (ReMA), and calibrates stuff mask logits and re-scores thing candidates with a within-scene road reference (ReCal). It reports the best PQ on all intra-nuScenes and cross-dataset targets.

Critical strengths

The strengths of the paper are listed as follows:

  • Well written and easy to follow. Section 3.1 states the task formally, Section 3.2 previews ReMA as four steps before giving Eq. 1 to 6, Fig. 2 (left) mirrors that order, and Table 1 separates LiDAR-only, image-only, and multimodal rows by color. A reader can locate every design choice quickly.
  • Lightweight design with cost reporting. Token-to-patch gates reuse the attention maps, the router is zero-initialized to neutral ω = 1, and ReMA plus ReCal add 6.62M parameters and 19.7 ms over the frozen-DINOv2 variant. Ablations show monotone nighttime gains from 42.08 to 49.34 to 52.06 PQ.
  • New DG protocol for mm-3DPS. The protocol trains on Boston sunny-daytime nuScenes and tests on nighttime, rainy, Singapore, SemanticKITTI, and Waymo, with documented label maps and frame counts.
Critical weaknesses

The weaknesses of the paper are listed as follows:

  • Compositional novelty. ReMA joins REIN tokens [1], geometry-conditioned tokens as in DepthForge [2], and reliability routing as in MoME [3]; ReCal re-scoring follows Mask Scoring R-CNN [4] and SPFormer [5].
  • Cross-dataset claims lack ablation. Tables 2 and 3 cover only intra-nuScenes, so whether the 20-point SemanticKITTI gain stems from ReMA gating, GridMask region dropout, or ReCal is unknown. SemanticKITTI is evaluated on sequence 07 only (1,101 frames), the xMUDA validation split [6], not test sequence 08 (4,071 frames).
  • Missing baselines and competitor costs. Only four adapted methods appear. Missing are UP-Fuse [7], a reliability-aware mm-3DPS method, the multimodal DG methods BEV-DG [8] and [9], and the LiDAR DG method LiDOG [10]. Table 6 gives parameters and latency only for the authors’ variants.

[1] Wei, Zhixiang, et al. "Stronger, fewer, & superior: Harnessing vision foundation models for domain generalized semantic segmentation." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[2] Chen, Siyu, et al. "Stronger, steadier & superior: Geometric consistency in depth vfm forges domain generalized semantic segmentation." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025.

[3] Park, Konyul, et al. "Resilient sensor fusion under adverse sensor failures via multi-modal expert fusion." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

[4] Huang, Zhaojin, et al. "Mask scoring r-cnn." 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019.

[5] Sun, Jiahao, et al. "Superpoint transformer for 3d scene instance segmentation." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 2. 2023.

[6] Jaritz, Maximilian, et al. "xmuda: Cross-modal unsupervised domain adaptation for 3d semantic segmentation." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[7] Mohan, Rohit, et al. "Up-fuse: Uncertainty-guided lidar-camera fusion for 3d panoptic segmentation." arXiv preprint arXiv:2602.19349 (2026).

[8] Li, Miaoyu, et al. "Bev-dg: Cross-modal learning under bird’s-eye view for domain generalization of 3d semantic segmentation." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

[9] Li, Fan, et al. "No Object Is an Island: Enhancing 3D Semantic Segmentation Generalization with Diffusion Models." Advances in Neural Information Processing Systems 38 (2026): 75836-75857.

[10] Saltori, Cristiano, et al. "Walking your lidog: A journey through multiple domains for lidar semantic segmentation." Proceedings of the IEEE/CVF international conference on computer vision. 2023.

Questions for authors

It would be great if the authors could focus on my concerns in the Weaknesses section in the rebuttal.

3 comments
AuthorsOpenReview ↗

Response to W1: Methodological Novelty

W1. Novelty Beyond Existing Methods.

We respectfully disagree. While token-based VFM adaptation, reliability routing, and confidence re-scoring have been explored individually, our novelty lies in designing domain-generalizable mechanisms specifically for multimodal 3D panoptic segmentation (mm-3DPS). This setting introduces two distinct challenges: (i) determining where and to what extent each sensor should be trusted under unseen domain shifts, and (ii) calibrating stuff and thing predictions whose reliability deteriorates differently. Our contributions are not straightforward combinations of existing techniques, as detailed below.

  1. vs. REIN [1] and DepthForge [2]. Both methods adapt VFMs using a single RGB image, with DepthForge deriving geometric information by depth estimated from that same image. In contrast, our setting involves two independent sensors (RGB and LiDAR), whose reliability can vary substantially across regions and domains. Simply applying REIN to the image or multimodal branch does not address this challenge. Indeed, under nighttime conditions, both variants underperform the LiDAR-only branch by 2.97 and 24.64 on PQ, respectively, highlighting the need for explicit cross-modal reliability modeling.
  2. vs. MoME [3]. MoME addresses sensor failures through hard selection (argmax) among LiDAR, camera, and fusion expert decoders for each object query. While effective for severe sensor failures, this approach does not explicitly capture the continuous, spatially varying reliability changes caused by unseen domain shifts. In contrast, ReMA independently and continuously modulates RGB and LiDAR feature reliability, enabling fine-grained, region-adaptive multimodal representations rather than discrete expert selection.
  3. vs. Mask Scoring R-CNN [4] and SPFormer [5]. These methods estimate mask quality to refine instance confidence but do not explicitly address domain shifts that disrupt the relationship between predicted confidence and actual segmentation quality. ReCal instead introduces frame-specific, road-referenced descriptors, combining relative features with semantic and mask consistency for re-scoring. By anchoring calibration to the current scene, ReCal aims to reduce sensitivity to domain-dependent feature shifts, yielding consistent improvements across unseen target domains.

In summary, ReMA and ReCal are designed to address complementary challenges specific to DG mm-3DPS: spatially varying cross-modal reliability and domain-robust instance calibration. Their contributions lie in these problem-specific mechanisms.

AuthorsOpenReview ↗

Response to W2: Cross-Dataset Ablations and Evaluation Protocol

W2.1. Cross-dataset Component Ablation.

We thank the reviewer for this suggestion and provide additional cross-dataset ablations in Table R1. In ReMA, reliability gating is the core mechanism, while region-wise modality dropout serves as a complementary training strategy that exposes the model to varying local modality degradation.

Compared with the original base model, introducing a frozen VFM improves SemanticKITTI/Waymo PQ by 10.05/6.24 points. Building on this VFM baseline, ReMA gating alone yields +4.68/+6.53 PQ, GridMask contributes +3.40/+2.37 PQ, and ReCal provides +6.64/+10.10 PQ. Combining these components, full RAMP3D achieves the strongest cross-dataset performance, with overall gains of 20.28/23.65 PQ over the base model.

Importantly, ReMA is not architecturally tied to a dropout strategy. We additionally evaluate Random Erasing [11] as an alternative region-dropout strategy, demonstrating the flexibility of the training framework to accommodate different masking operators.

Table R1. Cross-dataset ablation of RAMP3D components. Italicized indicate the source domain.

MethodReMA GatingReMA: Region DropoutReCalnuScenes PQsKITTI PQWaymo PQ
Original IAL–––79.6325.8520.15
Frozen VFM baseline–––80.7035.9026.39
+ ReMA Gating✓––80.9040.5832.92
+ Region Dropout–GridMask–80.3339.3028.76
+ Region Dropout–Random Erasing–79.9540.6627.65
+ ReCal––✓80.0742.5436.49
Full RAMP3D✓GridMask✓82.1846.1343.80

W2.2. SemanticKITTI Evaluation Protocol.

In this paper, RAMP3D and all competing methods are evaluated using their final training checkpoints, following common target-independent DG evaluation practice [12–14].

xMUDA uses Sequence 07 for checkpoint selection and Sequence 08 for testing. In contrast, our protocol has no selection step, so the two sequences play the same role as unseen target data, and we reported sequence 07. We additionally evaluate RAMP3D on sequence 08 with the same checkpoint. The nearly identical PQ on the two sequences (46.13 vs. 46.10) shows that the overall panoptic generalization performance is consistent.

Table R2. RAMP3D evaluation on SemanticKITTI Sequences 07 and 08.

Sequence# FramesPQPQthPQst
071,10146.1350.0845.34
084,07146.1048.3045.66

[11] Zhong et al. Random Erasing Data Augmentation. AAAI, 2020.

[12] Zhao et al. Style-hallucinated dual consistency learning for domain generalized semantic segmentation. ECCV, 2022.

[13] Zhong et al. Adversarial style augmentation for domain generalized urban-scene segmentation. NeurIPS, 2022.

[14] Termöhlen et al. A re-parameterized vision transformer (ReVT) for domain-generalized semantic segmentation. ICCVW, 2023.

AuthorsOpenReview ↗

Response to W3: Missing Baselines and Competitor Costs

W3.1. Missing Baselines.

We adapt UP-Fuse [7], BEV-DG [8], and LiDOG [10] to the source-only mm-3DPS DG setting using the same backbones and the unified evaluation protocol described in Sec. 4.2. UP-Fuse's linked repository provides only the Panoptic-Waymo toolkit, without model code. We therefore reproduce its uncertainty fusion and cross-modal interaction modules within IAL. We have added all three adapted methods to Table 1 of the revised manuscript, where the full results are reported. We also attempted to reproduce XDiff3D [9], but encountered missing implementations of key modules and configuration parameters in the released code, preventing faithful reproduction.

Table R3. Comparison with additional DG baselines. Italicized columns indicate source-domain results.

MethodModalityVFMUS S.D.NighttimeRainySingaporenuScenessKITTIWaymo
IAL3DL-74.9946.98 68.30 57.07 80.99 40.1035.95
IAL3D + LiDOGL-73.1246.2165.9555.9580.9033.5423.11
IAL + BEV-DGL+CDINOv280.0144.4274.8567.6777.7129.2726.49
IAL + UP-FuseL+CDINOv279.4946.1571.4264.7380.3136.5430.19
RAMP3DL+CDINOv280.1252.0677.6569.0282.1846.1343.80

Table R3 reveals the limitations of transferring existing methods to mm-3DPS DG. None of the three adapted methods surpasses the LiDAR-only IAL3D baseline on Nighttime, SemanticKITTI, or Waymo, whereas RAMP3D exceeds it by 5.08/6.03/7.85 PQ, respectively. These results highlight the following limitations:

  • LiDOG [10] primarily addresses LiDAR sensor shifts through BEV semantic supervision. However, BEV projection may introduce class ambiguity and lose fine-grained information, particularly for small objects, while lacking instance-aware supervision. Consistently, LiDOG reduces PQ across all target domains compared with IAL3D.

  • BEV-DG [8] assumes both modalities are equally trustworthy. This assumption fails when modalities face degradations, like it falls below LiDAR-only by 2.56 PQ at night and by 10.83 / 9.46 on SemanticKITTI / Waymo, where camera coverage and LiDAR pattern differ.

  • UP-Fuse [7] learns visual uncertainty from synthetic image corruptions, showing relatively strong performance under nighttime degradation but limited gains under other domain shifts. This suggests that predefined visual corruptions may not adequately capture complex, mixed, and high-level semantic or geometric shifts.

    We further compare with the day-to-night results reported in UP-Fuse's original Table VI. RAMP3D achieves +6.4 PQ, including +12.4 PQ-th and +0.3 PQ-st, with the improvement primarily driven by stronger thing segmentation under nighttime shifts.

Table R4. Day-to-night comparison with UP-Fuse. Following the protocol of [7], models are trained on daytime scenes and evaluated on nighttime scenes, and results are averaged over the 12 classes present at night.

PQPQ-thPQ-st
UP-Fuse64.364.963.7
Ours70.777.364.0

RAMP3D estimates the reliability of both modalities per region and calibrates the panoptic outputs with scene-relative features, surpassing all compared methods on all five targets.

W3.2. Computational Cost Comparison.

We thank the reviewer for this suggestion. We have extended the computational cost comparison to all adapted baselines, reporting parameter counts and inference latency in Table R5.

Compared with the frozen DINOv2 baseline, RAMP3D introduces only 6.62M additional parameters (+4.5%) and 10.2 ms latency (+7.8%). Compared with UP-Fuse, it adds just 9.1 ms (+6.9%) while achieving substantially better cross-domain performance. These results demonstrate that RAMP3D achieves stronger generalization with modest computational overhead. We have incorporated the complete comparison into Appendix Sec. C of the revised manuscript.

Table R5. Model complexity and inference latency. We measure all methods on an RTX 5090 with batch size 1 under same settings.

MethodVFMTotal params (M)Trainable (M)Latency (ms)
3D (LiDAR-only)
IAL₃D-51.4751.4753.4
+ LiDOG-52.3152.3153.6
+ DCGL-51.4751.4753.4
2D (image-only)
IAL₂D-26.9626.9637.8
+ REINDINOv2111.7225.5396.6
Multi-modal
IAL (Baseline)-64.8764.8779.4
+ DCGL-64.8864.8879.4
+ xMUDA-64.8864.8882.1
+ VFMDINOv2145.5959.41130.5
+ REINDINOv2151.2165.03134.2
+ BEV-DGDINOv2146.1459.95131.5
+ UniDSegCLIP142.3855.51238.9
+ Up-FuseDINOv2146.3560.17131.6
+ ReMADINOv2152.1065.91136.0
+ ReMA + ReCal (RAMP3D)DINOv2152.2166.02140.7

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.