acceptodds
Under review as a conference paper at ICLR 2027

Dual-diffusion Spatial Interaction Representation Generalization for 3D Semantic Segmentation

Abstract

3D semantic segmentation in unknown environments is crucial. Current methods utilize cross-modal learning and introduce 2D semantic priors to improve domain-generalized 3D segmentation. However, the heterogeneous representations of different modalities hinder cross-modal interaction, limiting the adaptation of 2D semantics to 3D space. In this paper, Dual-diffusion Spatial Interaction Representation (DSIR) is proposed to explore how cross-modal diffusion representations enhance 3D spatial semantics to improve the generalization of 3D semantic segmentation. Specifically, Spatial Representation Alignment (SRA) is first proposed to align the cross-modal diffusion representations, which designs manifold alignment of attributes and spatial coordinates to promote discriminativeness and spatial consistency before modalities interact. Secondly, Bidirectional Interaction Representation (BIR) models dual-diffusion interactions, which designs spatial joint adaptive representations to enable 2D and 3D bidirectional interactions. Finally, Spatial Semantic Modulation (SSM) is proposed to enhance 3D spatial semantics through effective interaction representations, which modulates local semantics in 3D space while suppressing spatially uncorrelated interaction noise, thereby improving the generalization of 3D semantic segmentation. Extensive experiments demonstrate that the proposed method has state-of-the-art performance. The code will be available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
3D point cloud semantic segmentation, Domain generalization, Cross-modal representation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DxViOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work develops diffusion features for domain-generalized 3D semantic segmentation. SRA selects and spatially aligns diffusion activations, BIR exchanges information between modalities at corresponding points, and SSM injects the resulting features into the segmentation branches. Experiments cover four domain shifts and report improvements in ensemble segmentation accuracy over the comparative methods.

Critical strengths

The main results are promising. In Table 1, the DG ensemble outperforms XDiff3D for most settings. The paper includes useful component ablations and parameter studies. Table 3 shows clear gains from adding BIR and SSM on two tasks, while Tables 4,5,6, and 7 examine diffusion variants, fusion methods, timesteps, and dimensions.

Critical weaknesses
  1. The specific benefits of the proposed mechanisms need stronger controls. Table 3 introduces two diffusion priors together with SRA, so SRA’s contribution is unclear.

  2. The comparison with recent work is limited, particularly for DG, the main focus of this paper. Table 1 includes two DA methods from 2025, but only one DG method from 2025. Please include additional relevant methods from 2025 and/or 2026 under a comparable evaluation protocol.

Questions for authors
  1. The baseline results are inconsistent across tables. For vKITTI→sKITTI on DG, Table 2 reports UniDSeg’s 3D mIoU as 42.12 and Ours 3D mIoU as 59.64, consistent with their listed class scores, whereas Table 1 reports 44.7 for UniDSeg and 59.7 for Ours. The inconsistent baseline results need to be resolved.

  2. Figure 2 covers four domain shifts but shows only one example per setting and compares only with UniDSeg. Please include more diverse scenes and zoomed-in comparisons, particularly for small objects, class boundaries, and sparse regions. Adding XDiff3D would also help show where the proposed method improves over the closest baseline.

2 comments
AuthorsOpenReview ↗

Additional experiments are provided to compare the advantages of the method. More comparisons on recent work.

W1. Isolating the contribution of SRA.

W1.A. Thank you for your suggestion. The table below supplements the individual contribution of SRA (as modified in Table 3 of the paper). The fourth and fifth data rows in the table below correspond to the variants without and with SRA, respectively. To achieve the variant without SRA, we average each layer's residual activations within each frame for each modality and assign the resulting features to all valid points in that frame. In 3D results,adding SRA improves 2.3% (43.7% to 46.0%) on vKITTI/sKITTI and 0.2% (64.7% to 64.9%) on nuScenes:USA/Sing. In xM results,adding SRA improves 1.8% (59.5% to 61.3%) on vKITTI/sKITTI and 0.3% (72.1% to 72.4%) on nuScenes:USA/Sing. The experiments demonstrate the effectiveness of the proposed SRA.

3DDM2DDMSRABIRSSMvKITTI/sKITTI (2D)vKITTI/sKITTI (3D)vKITTI/sKITTI (xM)nuScenes:USA/Sing (2D)nuScenes:USA/Sing (3D)nuScenes:USA/Sing (xM)
-----57.644.760.066.564.572.3
✓----57.933.060.966.364.471.8
-✓---57.943.262.067.164.772.6
✓✓---57.843.759.565.864.772.1
✓✓✓--57.446.061.366.464.972.4
✓✓✓✓-58.058.763.867.869.473.5
✓✓✓✓✓60.159.665.470.970.875.0

W2. Comparisons with recent work.

W2.A1. Thank you for your suggestion. We have searched for all relevant papers again on Google Scholar. The comparisons that can be added are shown in the table below. Regarding the DA method, UniDxMD [1] (2025 ICCV) reports the same validation settings in the appendix. For MS-xMUDA [2] (2025 KSUCIC), we report the pseudo-label retraining results and use the nuScenes-Lidarseg columns with point-level annotations for both nuScenes transfers. UniD-Shift [3] (2026 arXiv) reports 74.5 xM on USA → Singapore. However, the paper leaves its use of unlabeled target training data and its exact test split unspecified. We will add a matched-protocol comparison with UniD-Shift after the camera-ready submission. Regarding the DG method, BeyondSparse [4] (2026 AAAI), DGLSS++ [5] (2026 TPAMI), GRCNet [6] (2026 IJCV) are proposed for LiDAR-only setting, which differs from the cross-modal method. To the best of our knowledge, no other existing cross-modal domain generalization methods for the RGB-LiDAR modality follow this same experimental protocol. Extending these LiDAR-only methods to the RGB-LiDAR fusion (xM) scenario would necessitate the integration of an RGB branch. Since these LiDAR-only methods rely on the LiDAR SemanticSTF dataset, our future work could explore how to leverage single 3D diffusion models to enhance domain generalization.

Method\xM mIoU (%)USA → SingaporeDay → NightvKITTI → SemanticKITTIA2D2 → SemanticKITTI
UniDxMD [1] (2025 ICCV)74.374.663.357.0
MS-xMUDA [2] (2025 KSUCIC)69.562.7—54.1
UniD-Shift [3] (2026 arXiv)74.5———
Ours74.872.669.860.3

W2.A2. Currently, there has been insufficient exploration of cross-modal domain generalization methods within the field. Our complete code and model weights will be publicly available on GitHub after the paper is accepted (as stated in the abstract), facilitating further research on this cross-modal domain generalization models.

AuthorsOpenReview ↗

Other Response.

Q1. Different numbers between Tables 1 and 2.

Q1.A. Thank you for checking the class averages. The differences arise because our reproduction differs from the original paper in UniDSeg [7], and separate training runs of D²SIR. The 44.7 in Table 1 is the 3D mIoU reported in the original UniDSeg paper for vKITTI → SemanticKITTI DG. The 42.12 in Table 2 comes from our reproduction of the ViT-L/14 checkpoint at 39k iterations. Therefore, we have highlighted the reproduction values ​​in red in Table 2. For D²SIR, the original Table 1 run yielded 59.67, which rounds to 59.7. The rerun used for Table 2 and subsequent experiments yielded 59.64, which rounds to 59.6. We have standardized the manuscript on the rerun, reporting 59.6 at one decimal place and 59.64 at two. Thanks again for the reminder.

Q2. Qualitative comparisons in Figure 2.

Q2.A. More comparisons have been added to Figure 2, including other methods XDiff3D [8], and zoomed-in comparisons. Additionally, to expand the comparison, we added two more frames per setting to the appendix, with full views and zoomed-in regions that highlight small objects, class boundaries, and sparse regions.

References

[1] Liang et al. UniDxMD: Towards Unified Representation for Cross-Modal Unsupervised Domain Adaptation in 3D Semantic Segmentation. ICCV, 2025. Paper · Supplement

[2] Sun et al. Cross-modal unsupervised domain adaptation for 3D semantic segmentation via multi-scale fusion-then-distillation. Journal of King Saud University Computer and Information Sciences, 2025. Paper

[3] Zhang et al. UniD-Shift: Towards Unified Semantic Segmentation via Interpretable Share-Private Multimodal Decomposition. arXiv:2605.07356, 2026. Paper

[4] Wu et al. BeyondSparse: Facilitating Mamba to Enhance Cross-Domain 3D Semantic Segmentation in Adverse Weather. AAAI, 2026. Paper

[5] Kim et al. Toward Enhanced Representation Learning for Single-Source Domain Generalization in LiDAR Semantic Segmentation. IEEE TPAMI, 2026. Paper

[6] Yang et al. Explicit Geometry-Reflectance Domain Shift Modeling for Robust LiDAR Segmentation in Adverse Weather. IJCV, 2026. Paper

[7] Wu et al. UniDSeg: Unified Cross-Domain 3D Semantic Segmentation via Visual Foundation Models Prior. NeurIPS, 2024. Paper

[8] Li et al. No Object Is an Island: Enhancing 3D Semantic Segmentation Generalization with Diffusion Models. NeurIPS, 2025. Paper

Reviewer_LwTjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

D²SIR addresses domain generalization for 3D semantic segmentation using paired images and point clouds. It combines frozen image diffusion features with LiDAR diffusion features extracted from range maps. SRA selects diffusion activations and samples them at corresponding 3D points. BIR performs bidirectional interaction between the modality tokens at each point. SSM aggregates the interaction features and applies learned modulation to the segmentation branches at covered locations. The stated training objective combines source supervision with prediction consistency between branches. The authors evaluate four driving data transfer settings and report ensemble improvements over the listed baselines. Component ablations and fusion comparisons further assess the proposed integration pipeline.

Critical strengths
  1. Shared 3D point indices provide explicit correspondence between image and LiDAR diffusion features before interaction. Spatial aggregation then connects these representations to the segmentation branches. The coverage rule preserves features at locations without valid conditioning information. This provides a coherent framework for incorporating generative geometry priors into multimodal segmentation and extends the use of image diffusion representations alone.
  2. Table 1 reports ensemble improvements across all four domain generalization settings. The evaluations cover geographic and illumination changes as well as transfer from synthetic to real data. Table 3 also shows gains from adding BIR and SSM to the dual diffusion configuration. These findings support the practical value of the complete framework and provide evidence for its integration modules.
Critical weaknesses
  1. The introduction attributes limited cross-modal interaction partly to architectural differences and motivates dual diffusion through architectural similarity. However, the experiments do not establish why this similarity improves generalization. The single prior variants in Table 3 also omit processing stages used by the full pipeline. Their comparison therefore does not isolate the benefit of combining two priors under comparable processing capacity. This limits the evidence for the paper’s distinctive dual diffusion contribution.
  2. Section 3.2 treats matching activation roles as attribute alignment. This defines a feature selection strategy but does not demonstrate its effect on compatibility between the two representations. The learned projections in Eq. (7) could also improve compatibility. The analysis does not separate their contribution from activation selection and coordinate correspondence. Consequently, the specific benefit of the proposed attribute alignment remains unclear.
  3. Section 3.3 uses the distinction between conditional and joint distributions to motivate BIR over cross-attention. The attention operation in Eq. (9) does not by itself establish this probabilistic distinction. Table 3 supports the usefulness of BIR but does not validate the proposed joint distribution interpretation. Bidirectional exchange is clearly implemented. The stronger explanation for its advantage remains insufficiently substantiated.
  4. Section 3.4 combines spatial aggregation with coverage control and learned modulation in SSM. The coverage indicator records feature availability rather than semantic reliability. Learned modulation could still filter unhelpful information. However, Table 5 evaluates the overall fusion design without isolating this filtering effect. The results support the effectiveness of SSM more directly than the claimed suppression of spatially unrelated interaction noise.
Questions for authors

None.

2 comments
AuthorsOpenReview ↗

Additional experiments have been included to demonstrate the proposed method. The descriptions in the manuscript have also been revised accordingly.

W1. The contribution of dual diffusion.

W1.A1. Thank you for your suggestion. The table below supplements the contribution of dual diffusion (as modified in Table 3 of the paper). As shown in the newly added dual diffusion line, combining two diffusion models improves 3D results compared to single 3D diffusion models: dual diffusion improves 10.7% (33.0% to 43.7%) on vKITTI/sKITTI and 0.3% (64.4% to 64.7%) on nuScenes:USA/Sing in 3D results. However, the fusion results (xM) all show a decrease. This is because diffusion fusion with similar architectures requires further representation alignment. Therefore, the proposed modules design the “Alignment-Interaction-Enhancement” process to progressively leverage dual diffusion representations for enhancing 3D semantic generalization. Experiments demonstrate the effectiveness of the proposed method for cross-modal interactions.

3DDM2DDMSRABIRSSMvKITTI/sKITTI (2D)vKITTI/sKITTI (3D)vKITTI/sKITTI (xM)nuScenes:USA/Sing (2D)nuScenes:USA/Sing (3D)nuScenes:USA/Sing (xM)
-----57.644.760.066.564.572.3
✓----57.933.060.966.364.471.8
-✓---57.943.262.067.164.772.6
✓✓---57.843.759.565.864.772.1
✓✓✓--57.446.061.366.464.972.4
✓✓✓✓-58.058.763.867.869.473.5
✓✓✓✓✓60.159.665.470.970.875.0

W1.A2. Furthermore, the table below supplements the additional ablation experiments to further validate the contribution of diffusion. Firstly, we remove 3DDM while retaining the image prior and subsequent processes, which reduces both 3D and ensemble mIoU. This illustrates the advantage of 3D diffusion features in improving generalization. Secondly, replace 3DDM with non-diffusion SCN features. We use the SCN backbone's bottleneck features as the geometric token in BIR and retain two-token interaction and subsequent modulation. In the same interaction framework, the LiDM diffusion features yield a higher ensemble score than the SCN bottleneck features, further demonstrating the advantages of the proposed method.

Setting2D mIoU (%)3D mIoU (%)xM mIoU (%)
Full D²SIR60.1359.6465.39
-Remove 3DDM (LiDM)57.9241.8160.04
-Replace 3DDM (LiDM) with SCN geometric features59.2259.2563.95

W2. Demonstrate compatibility and benefit of SRA.

W2.A1. Thank you for your suggestion. The table below supplements the experiments to further demonstrate the compatibility. We test activation selection, coordinate correspondence, and the learned projections in Eq. (7) separately on vKITTI → SemanticKITTI. (1) Activation selection. We keep pointwise correspondence and the learned projections fixed and replace the selected attention activations with residual-layer outputs of comparable channel width. This replacement reduces 2D and xM mIoU by 1.37% and 1.08%, and experiments prove the effect of demonstrate activation selection on compatibility between the two representations. (2) Coordinate correspondence. We retain the original activation selection, learned projections, and pointwise features and shuffle cross-modal point pairings within each frame. This replacement reduces 2D, 3D, and xM mIoU by 0.29%, 17.83%, and 6.13%, respectively. This experiment shows the effect of coordinate correspondence on compatibility between the two representations. (3) Learned projections. With the full SRA setup, the learned and frozen variants start from the same initialization for the corresponding linear projections in both branches. In the control, we freeze the weights and biases from the start of training. Replacing the learned projections in Eq. (7) reduces 2D, 3D, and xM mIoU by 2.02%, 3.00%, and 2.66% percentage points, respectively, demonstrating the effect on compatibility between the two representations.

Setting2D mIoU (%)3D mIoU (%)xM mIoU (%)
Full D²SIR60.1359.6465.39
Replace activation selection58.7660.4164.31
Replace coordinate correspondence59.8441.8159.26
Replace learned projections in Eq. (7)58.1156.6462.73
AuthorsOpenReview ↗

Additional experiments have been included to demonstrate the proposed method. The descriptions in the manuscript have also been revised accordingly.

W2.A2. The table below supplements the experiments to further demonstrate the benefit of SRA. Both configurations use SD and LiDM with BIR and SSM disabled. In 3D results,adding SRA improves 2.3% (43.7% to 46.0%) on vKITTI/sKITTI and 0.2% (64.7% to 64.9%) on nuScenes:USA/Sing. In xM results,adding SRA improves 1.8% (59.5% to 61.3%) on vKITTI/sKITTI and 0.3% (72.1% to 72.4%) on nuScenes:USA/Sing. The experiments demonstrate the effectiveness of the proposed SRA.

DG settingSRA2D mIoU (%)3D mIoU (%)xM mIoU (%)
vKITTI → SemanticKITTIWithout SRA57.843.759.5
vKITTI → SemanticKITTIWith SRA57.446.061.3
nuScenes USA → SingaporeWithout SRA65.864.772.1
nuScenes USA → SingaporeWith SRA66.464.972.4

W3. BIR and cross-attention.

W3.A. We thank the reviewer for this important clarification. We agree that the attention operation in Eq. (9) alone does not establish a formal distinction between conditional and joint distributions, and that Table 3 demonstrates the empirical usefulness of BIR rather than validating this probabilistic interpretation. Our intended point was that BIR explicitly constructs a shared spatial interaction representation and performs bidirectional information exchange between the 2D and 3D branches, whereas a conventional cross-attention formulation typically uses one branch to condition the representation of the other. Therefore, we have revised Section 3.3 to present the conditional–joint distinction as an intuitive motivation rather than a formal probabilistic claim, and to state more precisely that the advantage of BIR is supported by its explicit bidirectional exchange and shared spatial representation, together with the ablation results in Table 3. Relevant modifications in the paper are marked in red.

W4. Learned modulation in SSM.

W4.A. Thank you for your suggestion. We agree that the coverage indicator records feature availability rather than semantic reliability and learned modulation could still filter unhelpful information. Therefore, we have modified the coverage indicator and learned modulation explanations in Section 3.4 of the paper and marked them in red. For Table 5 in the paper (see table below), other fusion methods actually apply the coverage mask, which restricts fusion to covered locations. Therefore, Table 5 compares 3D Semantic Local Modulation (3DSLM) with the other fusion methods under a common coverage constraint. We have clarified this in Table 5 and marked it in red. For noise filtering, we agree with the review that the results support the effectiveness of SSM more directly than the claimed suppression of spatially unrelated interaction noise. Therefore, we have revised this claim in the paper and marked them in red, to better explain the advantages of our method.

Method2D mIoU (%)3D mIoU (%)xM mIoU (%)
Cross-Attention59.354.160.5
Concat-MLP58.256.963.2
Add60.955.963.3
Gated Residual58.058.763.8
3DSLM (Our modulation)60.159.665.4
Reviewer_LokGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cross-modal domain-generalized 3D semantic segmentation. It combines image diffusion features with a frozen LiDAR diffusion prior. SRA selects activation roles and samples both modalities at corresponding point indices, BIR performs bidirectional mixing of modality tokens at each point, and SSM aggregates these representations and writes conditional residual updates to segmentation features under a coverage mask. The contribution is representation transfer and fusion, rather than a new diffusion generator.

Critical strengths

The reported synthetic-to-real improvement is meaningful. For DG vKITTI to SemanticKITTI, the 3D result is 59.7 versus 46.9 for XDiff3D, while the ensemble improves from 61.3 to 65.4. Ensemble improvements over XDiff3D are also reported in the other three DG settings: +1.5, +0.5, and +1.4 points. These results deserve attention, subject to protocol and resource clarification.

The method explicitly handles missing modality coverage: BIR specifies zero-, one-, and two-token cases, and SSM leaves uncovered semantic entities unchanged. The component table is informative rather than uniformly positive: adding the 3D diffusion prior alone can degrade the 3D result, while later interaction substantially improves it. Model-version, fusion, timestep, and width studies provide additional empirical context.

Critical weaknesses

First, the central dual-prior contribution is not isolated. Table 3 does not remove LiDM while retaining the final interaction/modulation framework and comparable capacity. Adding the second diffusion prior and SRA together also prevents identifying SRA independently. Since LiDM introduces real KITTI-360 pretraining, the synthetic-to-real gain cannot be attributed entirely to the proposed interaction architecture. This external prior may be legitimate and useful; it needs to be separated from the architecture claim through matched controls.

Second, implementation and inference requirements are unclear in the supplied material. The 13-page PDF ends with references and does not contain the appendix repeatedly cited for important configurations. It does not clearly specify whether SD, LiDM, and RGB conditioning remain active when reporting the 3D branch at inference. The closest XDiff3D paper explicitly discards diffusion/query operations after training, so this distinction affects input and compute budgets. It would be inappropriate either to assume LiDAR-only inference from the branch name or to assume the proposed method definitely retains all generative branches.

Third, several theoretical interpretations are stronger than the operations establish. Role selection, concatenation, and projection do not by themselves establish manifold or semantic alignment. Two-token self-attention permits bidirectional exchange, but does not demonstrate a probabilistic joint distribution or prove that cross-attention is biased. Different points remain independent inside BIR, although upstream features can still contain context. Similarly, SRA/SSM validity masks establish geometric coverage, not semantic reliability or noise rejection. More precise terminology and matched bidirectional-fusion and correspondence-perturbation controls would strengthen the argument.

Fourth, the reported UniDSeg baseline is inconsistent across tables. The six DG class scores in Table 2 average to 42.12, whereas Tables 1 and 3 report 44.7 for the same named setting; the original UniDSeg paper also reports 44.7. This requires clarification of checkpoint, split, label mapping, or table generation. The proposed method's 59.64/59.6/59.7 differences are much smaller and should not be treated as equally consequential.

Finally, the broad 3D-enhancement conclusion should distinguish the ensemble from individual branches: Day/Night DG 3D decreases from XDiff3D's 72.4 to 70.9 even though the ensemble improves. The validation source used to choose timesteps and width is also not specified, leaving strict DG selection unauditable without establishing that target leakage occurred.

Questions for authors

How does the final framework perform without LiDM, with a non-diffusion geometric encoder, and with comparable-capacity ordinary bidirectional fusion? Please provide the appendix, exact checkpoints and activation layers, source-only selection protocol, and a branch-by-branch inference graph with latency/memory. Please explain the UniDSeg baseline discrepancy and provide controlled evidence for coverage-specific noise suppression.

4 comments
AuthorsOpenReview ↗

Point-by-point responses to the weaknesses (W1–W5) and questions (Q1–Q4): More validation experiments and detailed explanations.

W1. Contributions of the dual diffusion priors, LiDM and SRA.

W1.A1. Thank you for your suggestion. The table below supplements the contribution of dual diffusion (as modified in Table 3 of the paper). As shown in the newly added dual diffusion line, combining two diffusion models improves 3D results compared to single 3D diffusion models: dual diffusion improves 10.7% (33.0% to 43.7%) on vKITTI/sKITTI and 0.3% (64.4% to 64.7%) on nuScenes:USA/Sing in 3D results. However, the fusion results (xM) all show a decrease. This is because diffusion fusion with similar architectures requires further representation alignment. Therefore, the proposed modules design the “Alignment-Interaction-Enhancement” process to progressively leverage dual diffusion representations for enhancing 3D semantic generalization. Experiments demonstrate the effectiveness of the proposed method for cross-modal interactions.

3DDM2DDMSRABIRSSMvKITTI/sKITTI (2D)vKITTI/sKITTI (3D)vKITTI/sKITTI (xM)nuScenes:USA/Sing (2D)nuScenes:USA/Sing (3D)nuScenes:USA/Sing (xM)
-----57.644.760.066.564.572.3
✓----57.933.060.966.364.471.8
-✓---57.943.262.067.164.772.6
✓✓---57.843.759.565.864.772.1
✓✓✓--57.446.061.366.464.972.4
✓✓✓✓-58.058.763.867.869.473.5
✓✓✓✓✓60.159.665.470.970.875.0

W1.A2. The table below supplements the experiments of Remove LiDM. We ran two controls for the LiDAR diffusion prior. Firstly, we remove LiDM while retaining the image prior and subsequent processes, which reduces both 3D and ensemble mIoU. This illustrates the advantage of 3D diffusion features in improving generalization. Secondly, replace LiDM with non-diffusion SCN features. We use bottleneck features from the SCN backbone trained with the segmentation model to replace the LiDM token in BIR. We keep the subsequent modules and retain two-token interaction. In the same interaction framework, the LiDM diffusion features yield a higher ensemble score than the SCN bottleneck features, further demonstrating the advantages of the proposed method.

Setting2D mIoU (%)3D mIoU (%)xM mIoU (%)
Full D²SIR60.1359.6465.39
-Remove LiDM (3DDM)57.9241.8160.04
-Replace LiDM (3DDM) with SCN geometric features59.2259.2563.95

W1.A3. The table below supplements the contributions of SRA. Both configurations use SD and LiDM with BIR and SSM disabled. In 3D results,adding SRA improves 2.3% (43.7% to 46.0%) on vKITTI/sKITTI and 0.2% (64.7% to 64.9%) on nuScenes:USA/Sing. In xM results,adding SRA improves 1.8% (59.5% to 61.3%) on vKITTI/sKITTI and 0.3% (72.1% to 72.4%) on nuScenes:USA/Sing. The experiments demonstrate the effectiveness of the proposed SRA.

DG settingSRA2D mIoU (%)3D mIoU (%)xM mIoU (%)
vKITTI → SemanticKITTIWithout SRA57.843.759.5
vKITTI → SemanticKITTIWith SRA57.446.061.3
nuScenes USA → SingaporeWithout SRA65.864.772.1
nuScenes USA → SingaporeWith SRA66.464.972.4

W2. Implementation details and inference costs.

W2.A1. Thank you for your suggestion. The table below provides more implementation details (as in the 8-page supplementary material). For vKITTI → SemanticKITTI, we use frozen SD v1.5 at timestep 50 with an empty text condition and frozen unconditional LiDM at timestep 150. The table lists the activation layers in concatenation order for each prior.

PriorActivation layerChannels
SD v1.5up_blocks.1.attentions.1.transformer_blocks.0.attn2.to_q1280
SD v1.5up_blocks.1.resnets.21280
SD v1.5up_blocks.2.attentions.1.transformer_blocks.0.attn2.to_q640
SD v1.5up_blocks.3.attentions.0.transformer_blocks.0.attn1.to_k320
LiDMoutput_blocks.4.1 self-attention query512
LiDMoutput_blocks.5.0 residual output512
LiDMoutput_blocks.7.1 self-attention query256
LiDMoutput_blocks.6.1 self-attention key256
AuthorsOpenReview ↗

Point-by-point responses to the weaknesses (W1–W5) and questions (Q1–Q4): More validation experiments and detailed explanations.

W2.A2. The table below supplements our method with a complexity analysis of XDiff3D and UniDSeg. When producing only the 3D prediction, our framework does not require the 2D segmentation backbone and head. Furthermore, diffusion models can be discarded after training to save compute budgets at inference. The table below compares the complexity of current methods (as in the appendix). Note that our reproduction of XDiff3D is based on its official code. However, its official code is incomplete. We achieved comparable performance on xM results based on the method described in the paper. However, the 3D results, no matter how hard we try, cannot reach comparable results. We will continue to contact the authors to determine the correct steps. Currently, according to the comparison in the table, our method achieves significant performance advantages while maintaining comparable computational costs. Furthermore, our complete code and model training weights will be publicly available on GitHub after the paper is accepted (as stated in the abstract), facilitating further research on lightweight domain generalization models.

MethodTrainable Params (M) ↓Train Step (ms) ↓Train Memory (GiB) ↓Infer. Latency (ms) ↓2D (mIoU) ↑3D (mIoU) ↑xM (mIoU) ↑
UniDSeg45.963584.7607.6511224.16357.644.760.0
XDiff3D†52.366849.50013.6851257.01958.828.056.3
D²SIR52.568722.27012.5611329.40960.159.665.4

W3. Interpretation of SRA, BIR, and SSM.

W3.A1. Thank you for your suggestion. The table below supplements the ablation experiments of SRA on vKITTI → SemanticKITTI. We ran separate controls for SRA's activation selection and pointwise correspondence. (1) Activation selection. We replace the selected attention activations with residual-layer outputs of comparable channel width and retain the correct point correspondence. This replacement reduces 2D and xM mIoU by 1.37% and 1.08%, and experiments prove the effect of demonstrate activation selection on compatibility between the two representations. (2) Correspondence perturbation. We keep the original activation selection, learned projections, and pointwise features and shuffle cross-modal point pairings within each frame. This replacement reduces 2D, 3D, and xM mIoU by 0.29%, 17.83% and 6.13%, respectively. This experiment shows the effect of coordinate correspondence is on compatibility between the two representations.

Setting2D mIoU (%)3D mIoU (%)xM mIoU (%)
Full D²SIR60.1359.6465.39
Conventional activation selection58.7660.4164.31
Correspondence perturbation59.8441.8159.26

W3.A2. The table below supplements the ablation experiments of BIR versus ordinary bidirectional cross-attention. We retain the same projections, attention blocks, and downstream modulation and change only the attention mask. In the control, each token attends only to the other modality's token at the same point. Full BIR lets each token attend to itself and to the other token. The two variants have the same parameter count. The experimental results show that full BIR improves 2D and xM by 0.36% and 0.62%, respectively. Bidirectional cross-attention scores 1.07% higher on 3D. We will add this control to the paper and describe BIR as pointwise bidirectional exchange. We will remove the interpretation in terms of a probabilistic joint distribution and inherent bias in cross-attention.

Setting2D mIoU (%)3D mIoU (%)xM mIoU (%)
Full D²SIR60.1359.6465.39
Bidirectional cross-attention59.7760.7164.77

W3.A3. SSM coverage and modulation. We agree with the review that the SSM coverage mask marks locations with geometric coverage. Therefore, we revised the description to focus on how SSM adapts cross-modal information to each branch's semantic representation and removed the noise suppression claim. Changes in Section 3.4 are marked in red. For Table 5 in the paper (see table below), other fusion methods actually apply the coverage mask. Therefore, Table 5 compares 3D Semantic Local Modulation (3DSLM) with the other fusion methods under a common coverage constraint. We have clarified this in Table 5 and marked it in red.

Method2D mIoU (%)3D mIoU (%)xM mIoU (%)
Cross-Attention59.354.160.5
Concat-MLP58.256.963.2
Add60.955.963.3
Gated Residual58.058.763.8
3DSLM (Our modulation)60.159.665.4
AuthorsOpenReview ↗

Point-by-point responses to the weaknesses (W1–W5) and questions (Q1–Q4): More validation experiments and detailed explanations.

W4. Different numbers between Tables 1 and 2.

W4.A. Thank you for checking the class averages. The differences arise because our reproduction differs from the UniDSeg paper, and separate training runs of D²SIR. The 44.7 in Table 1 is the 3D mIoU reported in the original UniDSeg paper for vKITTI → SemanticKITTI DG. The 42.12 in Table 2 comes from our reproduction of the ViT-L/14 checkpoint at 39k iterations. Therefore, we have highlighted the reproduction values ​​in red in Table 2. For D²SIR, the original Table 1 run yielded 59.67, which rounds to 59.7. The rerun used for Table 2 and subsequent experiments yielded 59.64, which rounds to 59.6. We have standardized the manuscript on the rerun, reporting 59.6 at one decimal place and 59.64 at two. Thanks again for the reminder.

W5. Day → Night and model selection.

W5.A1. For nuScenes Day → Night, the Quantitative Comparison section specifies that the state-of-the-art claim applies to the ensemble. The individual branches have separate results.

W5.A2. For model selection protocol, the table below supplements the validation setting. We follow the UniDSeg training protocol, with a change to the source cross-modal loss weight. UniDSeg uses different weights across the four settings, while we use 0.1 for all four. We update model parameters using source-domain training data only. We select hyperparameters, including loss weights, diffusion timesteps, feature width, and other training settings, on the target-domain validation sets. We use the final checkpoint for testing and reserve the target-domain test sets for final evaluation. The splits follow the official UniDSeg repository. DG configurations · SemanticKITTI splits · nuScenes splits

DG settingSource training setTarget validation setTarget test set
Virtual KITTI → SemanticKITTI5 scenes from Virtual KITTI train (2,126 frames)SemanticKITTI seq07 (1,101 frames)SemanticKITTI seq08 (4,071 frames)
A2D2 → SemanticKITTIA2D2 train (20 driving sequences, 27,695 frames)SemanticKITTI seq07 (1,101 frames)SemanticKITTI seq08 (4,071 frames)
nuScenes USA → SingaporeBoston scenes from the official nuScenes train split (15,695 frames)Singapore scenes held out from official train (2,770 frames)Singapore scenes from official val (2,929 frames)
nuScenes Day → NightDay scenes from the official nuScenes train split (24,745 frames)Night scenes held out from official train (606 frames)Night scenes from official val (602 frames)

Q1. Results without LiDM, with a non-diffusion geometric encoder, and with comparable-capacity bidirectional fusion.

Q1.A. Thank you for your suggestion. The table below supplements the ablation experiments. We evaluated the full model and three controls on seq08 for vKITTI → SemanticKITTI. (1) Remove LiDM. BIR receives only the SD token, which removes two-token interaction. Relative to the full model, 3D and xM drop by 17.83% and 5.35%, respectively. This control measures how much the full framework relies on the LiDM path. (2) Replace LiDM with a non-diffusion geometric encoder. We use SCN bottleneck features as the geometric token and retain two-token interaction and downstream modulation. Relative to this control, the full model gains 0.39% on 3D and 1.44% on xM. The larger gain is in the ensemble. (3) Use bidirectional cross-attention with the same parameter count. We retain the projections, attention blocks, and downstream modulation and change only the attention mask so each token attends only to the other modality's token at the same point. Full BIR gains 0.36% on 2D and 0.62% on xM. The cross-attention control scores 1.07% higher on 3D.

Setting2D mIoU (%)3D mIoU (%)xM mIoU (%)
Full D²SIR60.1359.6465.39
Remove LiDM57.9241.8160.04
Replace LiDM with SCN geometric features59.2259.2563.95
Bidirectional cross-attention with the same parameter count59.7760.7164.77
AuthorsOpenReview ↗

Point-by-point responses to the weaknesses (W1–W5) and questions (Q1–Q4): More validation experiments and detailed explanations.

Q2. Reproducibility details, model selection, and branch-by-branch inference costs (as response in W2 and W5).

Q2.A. Thank you for your suggestion. We updated the appendix with inference latency, memory usage, and an additional comparison with XDiff3D.

Q3. Different numbers in UniDSeg (as response in W4).

Q3.A. Thank you for your suggestion. The differences arise because our reproduction differs from the original paper in UniDSeg paper. The 44.7 in Table 1 is the 3D mIoU reported in the original UniDSeg paper for vKITTI → SemanticKITTI DG. To obtain per-class results for this setting, we reproduced UniDSeg using its training configuration. The 42.12 in Table 2 comes from our seq08 evaluation of the reproduced ViT-L/14 checkpoint at 39k iterations. We label 44.7 as the original paper's result and 42.12 as our reproduced result. We have clarified the reproduction in Table 2 and marked it in red.

Q4. Coverage mask and noise suppression (as response in W3).

Q4.A. Thank you for your suggestion. The SSM coverage mask indicates whether geometrically valid conditioning features are available at a location. And we agree with the review that SSM leaves features at uncovered locations unchanged, which makes modulation consistent with geometric coverage. We revised the description and removed the noise suppression claim. Therefore, we have modified the description in the paper and removed the noise suppression claim to better explain the advantages of our method. Relevant modifications in the paper are marked in red.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Reviewers' summary of our strengths

We thank all the reviewers and chairs for their time and feedback. Reviewers acknowledged D²SIR’s promising performance, module benefits, and useful experimental analysis. The table below outlines their specific comments:

Our strengthsSpecific comments in the reviewsReviewers
Promising results ✓“The reported synthetic-to-real improvement is meaningful.” — LokG; “Table 1 reports ensemble improvements across all four domain generalization settings.” — LwTj; “The main results are promising.” — DxViLokG, LwTj, DxVi
Gains from BIR and SSM ✓“Table 3 also shows gains from adding BIR and SSM to the dual diffusion configuration.” — LwTj; “Table 3 shows clear gains from adding BIR and SSM on two tasks.” — DxViLwTj, DxVi
Useful experimental analysis ✓“The component table is informative rather than uniformly positive.” — LokG; “The paper includes useful component ablations and parameter studies.” — DxViLokG, DxVi
Handling of missing coverage ✓“The method explicitly handles missing modality coverage.” — LokG; “The coverage rule preserves features at locations without valid conditioning information.” — LwTjLokG, LwTj
Explicit cross-modal correspondence ✓“Shared 3D point indices provide explicit correspondence between image and LiDAR diffusion features before interaction.” — LwTjLwTj
Coherent framework ✓“This provides a coherent framework for incorporating generative geometry priors into multimodal segmentation.” — LwTjLwTj
AuthorsOpenReview ↗

Summary of questions and answers in the rebuttal

The reviewers' concerns and our responses are summarized in the table below:

ConcernsReview commentsOur ActionsResponse details
Contribution of the LiDAR diffusion priorLokG (W1, Q1): “the central dual-prior contribution is not isolated.”; LwTj (W1): “Their comparison therefore does not isolate the benefit of combining two priors under comparable processing capacity.”Additional experimentsRemoving LiDM reduces ensemble mIoU from 65.39% to 60.04%. Replacing it with SCN geometric features while retaining two-token interaction and downstream modulation achieves 63.95%. These controls clarify the contribution of the LiDAR diffusion prior and its advantage over SCN features under the same interaction framework. ✓
Contribution of SRALokG (W1): “Adding the second diffusion prior and SRA together also prevents identifying SRA independently.”; LwTj (W2): “the specific benefit of the proposed attribute alignment remains unclear.”; DxVi (W1): “Table 3 introduces two diffusion priors together with SRA, so SRA’s contribution is unclear.”Additional experimentsWith both priors retained and BIR/SSM disabled, adding SRA improves 3D / ensemble mIoU by 2.3% / 1.8% on vKITTI→SemanticKITTI and 0.2% / 0.3% on USA→Singapore, isolating and supporting the contribution of SRA. ✓
Activation selection and pointwise correspondenceLokG (W3): “Role selection, concatenation, and projection do not by themselves establish manifold or semantic alignment.”; LwTj (W2): “The analysis does not separate their contribution from activation selection and coordinate correspondence.”Additional experimentsReplacing the selected activations while preserving pointwise correspondence reduces ensemble mIoU by 1.08%; perturbing pointwise correspondence while retaining activation selection reduces it by 6.13%. These separate controls support the contributions of activation selection and pointwise correspondence to representation compatibility. ✓
BIR comparison and interpretationLokG (W3, Q1): “Two-token self-attention permits bidirectional exchange, but does not demonstrate a probabilistic joint distribution or prove that cross-attention is biased.” The reviewer also requests “comparable-capacity ordinary bidirectional fusion.”; LwTj (W3): “Table 3 supports the usefulness of BIR but does not validate the proposed joint distribution interpretation.”Additional experiments and manuscript revisionCompared with bidirectional cross-attention with the same parameter count, BIR achieves 0.62% higher xM mIoU, supporting an ensemble-specific benefit. The revised BIR interpretation emphasizes pointwise bidirectional exchange rather than a formal joint-distribution claim. ✓
Implementation, inference costs, and evaluation protocolLokG (W2, W5, Q2): “implementation and inference requirements are unclear”; “validation source … is also not specified”; “distinguish the ensemble from individual branches”.Additional experiments and manuscript clarificationClarify implementation through model configurations and activation layers; clarify inference costs through experiments and complexity analysis; specify the evaluation protocol, including source-domain training, target-validation tuning, dataset splits, and ensemble-specific claims. ✓
Different numbers between Tables 1 and 2LokG (W4, Q3): “The six DG class scores in Table 2 average to 42.12, whereas Tables 1 and 3 report 44.7 for the same named setting”; DxVi (Q1): “The inconsistent baseline results need to be resolved.”Manuscript clarificationExplain the different numbers by identifying 44.7 as the original paper’s result and 42.12 as our reproduced result used for the per-class comparison. ✓

Our rebuttal provides additional evidence for the paper’s central contribution and responds to the reviewers’ comments point by point. We welcome further discussion and will respond promptly to any remaining questions. We sincerely thank the reviewers and chairs for their time and valuable feedback. If the reviewers find that their concerns have been satisfactorily addressed, we would be grateful if they would consider revising their scores upward. We also respectfully ask the Area Chair to consider the paper for acceptance in light of the additional evidence and clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.