acceptodds
Under review as a conference paper at ICLR 2027

Escaping the Texture Trap: Intervention-Guided Adaptive RGB–Geometry Fusion for Indoor 3D Segmentation

Abstract

Robust indoor RGB–geometry segmentation can fail when misleading appearance conflicts with valid geometric evidence. We introduce an intervention-guided framework that couples geometry-preserving texture interventions with input-adaptive fusion. Structured Texture Swapping (STS) creates paired clean and intervened observations while preserving labels and point correspondence. A Geometry–Texture Interaction Gated Unit (GT-IGU) predicts point- and channel-wise routing weights from ordered cross-modal evidence. Candidate-Oracle Routing Distillation (CORD) further supervises GT-IGU using label-aware, candidate-relative routing targets. Structure-Aware Texture-Swapping Invariance (SA-TSI) regularizes intervention-induced feature changes over compatible neighborhoods. All teacher and graph operations are training-only. On S3DIS and ScanNetV2, the system improves affected-region performance under synthetic texture conflicts across both backbones. On the reported appearance, geometry, and joint shifts, it improves full-scene mIoU over the evaluated RGB–geometry baselines. Ablations identify paired intervention–adaptive fusion as the primary driver, with CORD and SA-TSI providing further gains at near-baseline inference cost.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Semantic Segmentation, Multimodal Fusion, Routing Distillation, Texture Robustness

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2W3gOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates an important question: whether RGB–geometry segmentation models can maintain reliable predictions when RGB textures become misleading. To address this issue, the authors introduce texture interventions during training to improve segmentation robustness. The experimental results can show improvement on dealing with texture-change in different settings.

Critical strengths

The analysis of RGB texture interventions is meaningful, as existing segmentation models may over-rely on texture cues associated with specific semantic categories.

The experiments on unseen appearance shifts, such as illumination changes and color shifts, also provide valuable evidence of the method's robustness and generalization ability.

Critical weaknesses

My main concern is whether the proposed replacement-based texture intervention truly encourages the model to learn better RGB–geometry fusion, or simply makes it rely more heavily on geometry while reducing its dependence on RGB. If the latter is the case, the improvements may primarily come from suppressing texture information rather than learning genuinely adaptive multimodal fusion.

Questions for authors

Is cross-category texture replacement the most appropriate intervention strategy? Replacing an object's texture with that of another semantic category creates relatively artificial conflicts that may not accurately reflect real-world distribution shifts. Would more realistic perturbations, such as illumination changes, color shifts, or sensor-induced appearance variations, provide more effective training signals for robust multimodal fusion?

Does the proposed method truly learn adaptive RGB–geometry fusion, rather than simply increasing its reliance on geometry? I suggest adding a controlled experiment where RGB information remains reliable while geometry is corrupted, for example, through coordinate noise, point displacement, or partial point dropout. Comparing the baseline and the intervention-trained model under these conditions would help determine whether the proposed training strategy preserves the model's ability to leverage reliable RGB cues when geometric information becomes unreliable.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2W3g

Thank you for the constructive suggestions. We added analyses of RGB use and fusion weights, along with photometric-augmentation comparisons, and clarified the setting of the existing geometry-corruption experiments. Revisions are marked in green.

W1. Does the method learn adaptive fusion, or does it only rely more on geometry?

The model continues to use RGB when geometry degrades, and its fusion weights respond to which input is corrupted. The new analysis on ScanNetV2 with PointNeXt reports both weight responses and the effect of replacing RGB with the training-set mean color (Appendix F.3, Table 19; summarized in Sec. 4.4):

Evaluation settingMean geometry weight (ours)RGB-replacement drop (ours)RGB-replacement drop (baseline)
Clean0.514.126.18
External texture0.72——
RGB-C0.61——
Geometry corruption0.449.8710.24

Drops are in full-scene mIoU percentage points, measured with each model's trained parameters fixed. Geometry corruption averages noise, jitter, and dropout at severity 3. A dash indicates an unreported measurement.

The geometry weight decreases under geometry corruption and increases under appearance corruption. The RGB-replacement result confirms that RGB remains useful when geometry degrades. The response is also local: under a partial texture swap, the mean geometry weight rises from 0.52 to 0.84 inside the affected region and stays at 0.50–0.52 outside it (Table 9, Fig. 3).

Geometry-only performs better under whole-scene appearance corruption, including external texture (68.04 vs. 55.40). Our model performs better on clean data (+1.25 mIoU), under geometry corruption (+16.14 mIoU), and on object-level texture conflicts (+7.38 TR; Tables 2 and 19). These diagnostics support input-dependent fusion weights and continued use of RGB, alongside reduced RGB sensitivity on clean data.

Q2. A controlled experiment with reliable RGB and corrupted geometry.

This setting is evaluated in Tables 3 and 17: Gaussian coordinate noise, bounded jitter, and random point dropout alter geometry while leaving RGB intact. None of these corruptions is used in our training. Sec. 4.3 now states this setting explicitly (first paragraph, L. 363–364).

  • Our method outperforms the baseline in all 12 dataset–backbone–corruption settings at severity 3. Averaged over the three corruptions, the gains are 24.4 and 20.3 mIoU points on S3DIS with PointNeXt and PTv3, and 19.2 and 7.4 on ScanNetV2.
  • It also outperforms CorrAug, which is trained on these corruptions.
  • On ScanNetV2 with PointNeXt, our model reaches 66.50 mIoU, compared with 47.27 for the baseline, 50.36 for geometry-only, and 45.31 for RGB-only (Table 19).

These results demonstrate improved segmentation when geometry is corrupted and RGB remains intact. The RGB-replacement test in W1 confirms that RGB contributes to the predictions in this setting.

Q1. Is cross-category texture replacement the right intervention? Would realistic perturbations give a better training signal?

Photometric augmentation improves both models and provides additional gains when combined with our framework. Following your suggestion, Table 20 compares the baseline and our full framework, each with and without PhotoAug. This full-framework comparison tests complementarity with PhotoAug without isolating STS from the other components. PhotoAug applies random brightness, contrast, saturation, and hue shifts, plus random grayscale conversion, to the whole scene.

Training recipeCleanExternal textureConflict-region mIoURGB-C
Baseline70.7939.6014.8656.84
Baseline + PhotoAug71.4646.3326.5964.18†
Ours69.2955.4088.6463.88
Ours + PhotoAug70.8259.8090.6968.93†

All entries are mIoU (%) on ScanNetV2 with PointNeXt. Clean, External texture, and RGB-C use full scenes; conflict-region mIoU uses affected points and evaluated target categories. External texture uses projection strength alpha = 1.0. RGB-C averages all five severities of fog, brightness, and hue shifts. †Brightness and hue families overlap with PhotoAug training; fog is excluded. External texture is held out for all four configurations.

With the same PhotoAug recipe, our framework improves external-texture mIoU by 13.47 points and conflict-region mIoU by 64.10 points over the augmented baseline, with a 0.64-point clean-mIoU cost. Without PhotoAug, our framework also exceeds the augmented baseline on external texture and semantic conflicts; Baseline + PhotoAug is slightly higher on RGB-C.

For practical training, the results support combining PhotoAug with our framework: it improves both clean accuracy and robustness to held-out external texture. The swap metrics provide controlled conflict diagnostics, and full-scene mIoU confirms gains under the evaluated simulated shifts.

Reviewer_3Rz8OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

No generative AI was used to process the paper nor my review. This is it too keep the confidentiality of the work. Opinions of the reviewer are their own.

The paper studies robust indoor 3D segmentation. It focuses on two failure settings: dataset-level dependence on texture shortcuts and instance-level mislocation of fusion weights under appearance shift. For the first failure, authors maintain geometric structure of the objects while varying their color (a strategy called structured texture swapping, STS) and introduce Geometry-texture interaction gated unit (GT-IGU). A distillation strategy dubbed candidate-oracle routing distillation (CORD) evaluates fusion hypotheses on both the initial object and their augmented versions. Since the STS method might dramatically change the features of the network on the swapped regions, boundaries, authors also introduce structure-aware texture-swapping invariance (SA-TSI). SA-TSI grounds augmented instances to match the features of their original version.

Critical strengths
  • The idea of instance-level augmentation, via appearance variation has been studied before in 2D and is known to be effective.
  • The overall idea is sound.
  • Experiments under two datasets and two backbones offer a reasonable understanding of the phenomenon.
  • Results are appealing and almost always improve the performance of the base model the method is applied on during training. Nonetheless, I detail my concerns about the details of these results in the weaknesses section.
  • The efficiency limitations are clearly stated and are not unreasonable.
Critical weaknesses

Below are major concerns.

Method and experimental concerns

  1. Nature of variables. The variable vv is introduced L. 135 and used for indexing the features obtained from the shared encoder. However, there is no explanation as to why vv is introduced in the first place nor what it represents. I supposed and guessed that it stands for "views" and that up to tt views are generated. But this is extremely unclear and would benefit from being clarified. At L. 144, the Ω\Omega variables are introduced, but are not neatly defined. Please specify their dimensions. I could later infer from eq. 1 that they are indicator tensors of the same shape as the point cloud where values that fall in the object of interest are 1, else 0, but I am not confident about this and this should not be an effort from the reader. Similarly, gg and gˉ\bar{g} are never introduced at any point near their first appearance in equation 1, significantly impeding reading.
  2. Clarification of eq. 3. It is not clear why equation 3 is introduced when reading the text. Could authors clarify it and explain what and why they are doing what they are doing before writing the equation? ⊙\odot is not defined either, which is confusing. Why do authors use layer-normalized features? Are they simply hidden states from layer ll of the network? In that case, which layer?
  3. Clarification on the shared estimator. The shared estimator rϕr_\phi is introduced without ever being precisely described (L. 170). What instantiation do authors use, how many parameters, etc. This could be specified in sec. 4.1.
  4. Clarification of eq. 4. What is the [⋅,⋅][ \cdot, \cdot ] operator? I suppose it is concatenation but since concatenation has been introduced before as "Concat" (eq. 3) it is unclear.
  5. Hard to make sense of the intuition behind the graph. In the method section, L. 214, the graph structure is introduce in the distillation algorithm. Yet, it is hard for me to understand what it represents. What are its nodes and edges? Simply stating that is encodes spatial proximity and surface compatibility is not sufficient to clearly understand how it is created and what it actually represents. Generally speaking, I cannot understand this paragraph at all. What is the "raw preference target" (L. 214)? Why does that "operation suppresses isolated candidate decisions" (L. 215)? Is there some kind of pruning happening?
  6. The proposed method seem to negatively affect points outside the swap regions. Table 2 reports multiple augmentation baselines against the proposed approach on ScanNetV2 using PointNeXt. While the mIoU conf of the proposed approach significantly increase over the baselines, The mIoU (clean) of the scene decreases with respect to the baseline. This suggest that the method could negatively affect the areas outside the swapped regions.I wonder if authors have an explanation for this surprising result.
  7. Major concern related to the overall performance of the method. Although the presented results show that the proposed method consistently outperform the baseline, I have concerns about some of its components not being essential. For example, table 5 reports the full ablation and we can clearly see that, while STS works well in itself, adding the GT-IGU and SA-TSI components deteriorate the performance on the clean data. In particular, GT-IGU decreases the performances of 4.24 mIoU points on clean data with respect to the STS strategy. While I understand that TR and TDR increase on the other hand, the main capability of a segmentation network is to segment data usually obtained from clean sensors. Could authors comment on this result since there is only a reference to table 5 (L. 422), but no explicit analysis?
  8. Concerns about the metrics themselves. Following the previous question, while STS seems like an effective augmentation, there would be no context where the texture of two items are swapped in such way in real life. Therefore, it sounds to me that the TR, TDR and mIoU conf are not critical metrics to evaluate in-the-wild performance. While table 6 shows a similar trend, the performance drop between no CORD and full CORD on clean data is marginal, which seems to me like a much better sign.

Clarity concerns

  1. Paper clarity. Generally speaking, I find the paper hard to read. Many of the concepts are not properly introduced before authors employ them, the paper also uses a lot of jargon which impedes understanding.
  2. Clarify abstract. Abstract is the first exposure to the paper, and therefore should be self-contained. However, in the current form, L. 17 introduces the notion of "observation" which is unexplained, the introduction of the GT-IGU is fine but "routing weights from ordered cross-modal evidence" is extremely vague. What are the modalities in this case? Why are there routing weights that have not been introduced before? The second notion introduced at line 19 is CORD, but once again lacks clear explanation. It is not understandable at first reading what "label-aware, candidate-relative routing targets" are. This continues with line 22, where the abstract introduces graph operations without explanations. It makes sense there would be a teacher since distillation was introduced, but why are the authors mentioning about a graph without prior introduction? Consider something more explicit: "we use a graph for X, which, alongside the teacher are only leveraged during training time". The end of the abstract is a more legible. In that regard, L. 40 onward does a much better job at introducing the problem at hand. This kind of information would benefit form being added in the abstract in a clear manner.
  3. The paper seems conceptually misleading. The authors essentially introduce an augmentation strategy, "structured texture swapping", but never mention the word augmentation before L. 299 (discarding the related works). This is related to my earlier concerns about clarity of the paper. I do believe that there are a lot of concept confusions in the manuscript. Because objects are not properly and clearly defined as soon as they are mentioned the first time, this makes it hard to evaluate the proposed approach properly and challenge reading.
  4. Cross-modal confusion. I do not understand why the authors mention "cross-modal" (L. 18, 47, 51, 169, 181, 687). At no point the proposed approach is cross-modal since the only modality used is the point cloud itself. In the least conservative sense, it could be called adversarial, since the augmentations have the role of forcing the network to focus on the geometry of the object independently of its color. Perhaps the authors think of the notion of modality in a different aspect, but this needs clarification. Could authors please explain their definition of cross-modality and justify the reason they use this word/sentences?
  5. On the concept of "donor". In the experiment section, authors repeatedly use the word "donor". Without prior introduction. While I do understand the word itself, what is its meaning in this specific use case? Does it simply refer to source-target swaps?
  6. Figure 1 should be referenced more. I think that having this figure is essential in order to improve understanding. Since the method is quite complex, referencing the figure and location at which the process happens in the text might improve clarity of the paper and make the figure more useful.
  7. Figure 2 position. Figure 2 illustrates the idea developed in the paper clearly and would benefit from being placed earlier in the paper. Maybe during the introduction. It contains input and corruption and target, motivations (the baseline fails), and successful prediction of from the proposed method. It would help the reader to build a simple mental model of the manuscript earlier on.

From here on, I comment on minor concerns.

Minor weaknesses

  1. First sentence of introduction could be a bit more framed. L. 33 is very generic and would earn to be related more tightly to the actual problem at hand. Perhaps authors could directly mention about the problems 3D indoor segmentation face.
  2. Suggested reference. Authors might be interested in "Dataset Enhancement with Instance-Level Augmentations", O. Kupyn and C. Rupprecht, 2024 (ECCV), which aligns with their work (although in the 2D domain).
  3. Table format. Resize table 19 to fit in page. Resize table 12 to fit in page.
  4. Argmin fomatting. j∈Ωsrcj \in \Omega_{src} should be centered below argmin (L. 146).
  5. Meaning of I\mathbf{I}. I suppose that I\mathbf{I} at L. 150 is an indicator function? Could authors clarify this?
  6. Typo. Space between "alternative" and reference (L. 305).
  7. References issues. The "TransFusion" paper (Z. Bai et al., 2022), has a typo on one of its authors. It should be "Hongbo Fu" (L. 506), according to the CVF page. The Ego-exo4d citation is improperly formatted (cf. "et al. Malik") (L. 544). The "LaserMix" paper pages should be 21705-21715 instead of 21706–21716, according to the official CVF page. Please consider a full re-check of the references to avoid any such errors.
  8. Paragraph formatting. L. 461, there should be a new paragraph, as indicated by the bold text.
Questions for authors

The idea itself is interesting and the developed method seems sane since it re-uses well-known components of self-supervised learning (multi-view augmentations, teacher-student architecture, invariance via KL-divergence alignment), but only for regularization purposes, since authors still directly supervise the entire pipeline with the segmentation task signal from the GT. While the method is complex, relying on a lot of module and different types of supervision, I believe that the components themselves make sense and that the performance gain is expected and empirically verified. Specifically, STS and CORD seem like they are both effective at maintaining/improving performance of the original model while robustifying it.

Nonetheless, I have major concerns related to the necessity of GT-IGU and SA-TSI components since table 5 clearly shows that they hurt the performance on clean data. This aligns with the fact that their method is hard to understand. I wonder if these components are really needed in the first place. Moreover, there is no such context in real life where an STS-like corruption would be directly applied on data, meaning the presented approach is good for training, but evaluation of TR, TDR and mIoU conf do not seem important to me.

Moreover, the manuscript itself is extremely confusing and hard to read. Many notations and concepts are not properly introduced making the method extremely confusing, even after spending a substantial amount of time reading it. The paper also uses a lot of unnecessary jargon. Many sentences could be explained more clearly, and using concepts that were already developed in prior literature. Lots of concepts are recurrently used without proper introduction, impeding understanding.

I personally do not think this paper is ready to be accepted at this conference in the current state. I believe the manuscript would need to undergo substantial rewriting to become clearer. Especially on the method part. Experiments also need to clearly support claims. I do not believe that the ablations particularly emphasize the fact several components of the method are essential.

In the event where such fixes occur, I could consider changing my suggestion to borderline accept.

Please refer to the weaknesses section where I detail my concerns and questions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3Rz8 (Part 1 of 3): clean accuracy, components, and metrics

Thank you for the detailed review. We rewrote the abstract, clarified the method, and added analyses addressing your concerns. Relevant revisions are marked in blue. We address W6–W8 first, followed by clarity and minor points, using your numbering. All line references refer to the revised PDF.

W6. Does lower clean mIoU imply harm outside swapped regions?

Clean mIoU measures accuracy on unmodified scenes. To measure the swap's effect on surrounding points, we added a paired clean–intervened comparison restricted to points outside the swapped region, using ScanNetV2 with PointNeXt (Appendix D.4; Sec. 4.2, L. 367–369).

Outside-region diagnosticBaselineOurs
Paired mIoU drop ↓ (percentage points)7.430.26
Prediction flip rate ↓ (%)14.640.79

Both diagnostics show that our model substantially reduces the swap's effect on surrounding points. The paired inputs share the same geometry, labels, and evaluation points.

Table 3 reports clean accuracy separately: mIoU improves by 3.6/2.1 points on S3DIS with PointNeXt/PTv3 and by 0.4 on ScanNetV2 with PTv3; it decreases by 1.5 on ScanNetV2 with PointNeXt. W7 examines the corresponding component contributions.

W7. Contributions of GT-IGU and SA-TSI and the clean-accuracy cost.

Adding SA-TSI improves clean accuracy and every other metric in Table 5. The clean-accuracy decrease occurs at the preceding stage, where GT-IGU and the CORD pathway are introduced jointly. Sec. 4.5 now makes this distinction explicit (L. 454–463):

Cumulative configurationClean mIoU ↑TR ↑Conflict-region mIoU ↑
Baseline70.7933.5614.86
+ STS and paired consistency71.9725.1017.82
+ GT-IGU and CORD pathway67.7385.6786.34
+ SA-TSI (full model)69.2989.7588.64
Increment from adding SA-TSI+1.56+4.08+2.30

Results are on ScanNetV2 with PointNeXt (Table 5). Configurations are cumulative; the CORD pathway includes candidate exposure. Clean mIoU is measured on full scenes, while TR and conflict-region mIoU use the affected target regions. Scores are percentages; the final row gives increments in percentage points.

The table shows that STS alone improves clean accuracy but does not deliver the full model's conflict robustness. Jointly adding GT-IGU and CORD produces the largest robustness gain, with a 4.24-point clean-mIoU cost; SA-TSI then recovers 1.56 points. Adding SA-TSI also improves noise/jitter/dropout mIoU by 2.77/8.95/4.54 points, and boundary mIoU from 72.11 to 86.74 (Tables 5 and 10).

Two controlled comparisons further separate the contributions:

  • GT-IGU: Table 11 fixes the STS and SA-TSI objectives and disables candidate exposure and CORD. GT-IGU reaches 83.84 TR, compared with 61.48 for mean fusion and 72.31 for parameter-matched concatenation. It has the highest TR and conflict-region mIoU among these CORD-free designs, with a 1.15-point clean-mIoU cost relative to mean fusion (69.56 vs. 70.71).
  • CORD: with GT-IGU and SA-TSI fixed, adding the CORD pathway improves TR by 5.91 and conflict-region mIoU by 6.48 points, with a 0.27-point clean-mIoU cost (69.56 to 69.29; Table 6).

The gains extend beyond texture swaps: the joint GT-IGU + CORD stage improves noise/jitter/dropout mIoU by 12.64/16.59/24.47 points (Table 5). PhotoAug further narrows the clean gap to the equally augmented baseline to 0.64 points (70.82 vs. 71.46), while external-texture mIoU reaches 59.80 versus 46.33 (Table 20).

W8. Do the conflict metrics measure in-the-wild performance?

Our robustness evaluation includes full-scene mIoU under coordinate noise, jitter, dropout, external material projection, fog, brightness/hue shifts, and joint degradation (Tables 3 and 4). These shifts are absent from the original training recipe. At severity 3, our method outperforms the baseline in all 12 dataset–backbone–geometry-corruption settings; under external texture projection, mIoU is 55.40 versus 39.60.

TR, TDR, and conflict-region mIoU serve a separate purpose: they diagnose the response to a controlled texture conflict. Sec. 4.2 now states this role explicitly (L. 321–323). The full-scene results establish robustness beyond the texture-swapping operation used in training. They concern simulated shifts; evaluation under naturally occurring shifts remains future work.

AuthorsOpenReview ↗

Response to Reviewer 3Rz8 (Part 2 of 3): clarity and presentation

W9. Overall clarity. We rewrote the abstract and clarified the opening of Sec. 3 and Secs. 3.1, 3.3, and 3.4. We added definitions at first use for the notation you highlighted, and plain-language explanations precede the fusion evidence, candidate scoring, and graph refinement. The specific changes follow.

W1. Variables v, Omega, and local coordinates in Eq. 1.

  • v indexes two versions of the same scene: v = 0 is clean and v = t is texture-swapped (Sec. 3.1, L. 156–157).
  • Omega_tgt and Omega_src are point-index sets, each a category-constrained neighborhood of 128 points (Sec. 3.2, L. 169–170; Table 21). Omega_src is the donor patch. The affected set is Omega^(t) = Omega_tgt; the separate binary mask m_i is 1 for affected points and 0 otherwise.
  • The barred coordinates are obtained by independently centering and scaling each patch (L. 171–172). t_i and its tilde version denote point i's RGB before and after swapping.

W2. Purpose of Eq. 3, normalization, and feature location. GT-IGU assigns geometry/RGB weights per point and channel. Eq. 3 supplies evidence for this decision by concatenating the two features, their difference, squared difference, and element-wise product. Differences expose discrepancies; the product supplies multiplicative interactions. Argument order identifies the modality being scored by the shared network (Sec. 3.3, L. 192–209).

Parameter-free layer normalization reduces feature-scale differences when computing the weights. Eq. 4 fuses the original, unnormalized features.

For both PointNeXt and PTv3, fusion occurs after the final encoder stage and before the first decoder stage. Each modality tensor has shape N_f x 128, where N_f counts fusion-stage points and N counts input points. The fused features pass through the decoder and segmentation head; the backbone's skip connections are retained (L. 159–163).

W3. Shared estimator. The estimator is a two-layer MLP, 5D -> 64 -> D, with ReLU and D = 128. Its 49.34K parameters are shared across modalities (L. 161, 202–203). Table 11 compares its complexity with alternative fusion designs.

W4. Brackets in Eq. 4. The brackets stack score vectors along a new modality axis. Softmax operates over this axis, making the two weights sum to one at every point and channel (L. 203–204).

W5. CORD graph and refinement. The graph smooths point-wise geometry preferences across nearby points on compatible surfaces. Sec. 3.4 now explains its construction and purpose before introducing the refined target (L. 244–254); Appendix B.4 gives the weights and solver.

  • Raw target: each point's expected geometry coefficient under the softmax over candidate costs (Eq. 6). Independent point-wise estimates can be noisy.
  • Graph: nodes are affected points, each connected to its 16 nearest neighbors within that set. Edge weights depend on spatial distance and normal agreement, using 1 - |n_i dot n_j| for the normal term.
  • Refinement: a quadratic objective anchors values to their raw targets, more strongly at high confidence, and encourages agreement between strongly connected neighbors.
  • No pruning occurs. A low-confidence point can borrow its neighbors' preferences; weak affinities limit smoothing across geometric changes. This is what "suppressing isolated decisions" means.

W10. Self-contained abstract. The rewritten abstract (L. 13–32) states the problem of misleading appearance in indoor RGB–geometry segmentation and explains each component by its operation: STS copies RGB between patches while preserving geometry and labels; GT-IGU predicts fusion weights; CORD derives supervision from fixed mixtures evaluated against labels; and SA-TSI constrains feature changes under the swap. It also states that the teacher and auxiliary objectives are used only during training.

W11. STS as augmentation. STS is now explicitly called label-preserving data augmentation in the abstract, introduction, and related work (L. 16–17, 49, 132–133). "Intervention" is explained as changing RGB while preserving geometry and labels (L. 51–52).

W12. Meaning of cross-modal. We define the two modalities as per-point geometry (xyz) and appearance (RGB), processed by separate encoders and fused by GT-IGU. Both belong to one colored point cloud; "cross-modal" denotes interaction between these RGB-geometry streams (L. 39–40, 145–146).

W13. Donor. The donor is the source patch whose RGB values are copied to the target patch, now defined at first use (L. 49–50, 170).

W14. Figure 1 references. References now appear in the introduction (L. 90–91), the opening of Sec. 3 (L. 142), and the STS, GT-IGU, CORD, and SA-TSI subsections.

W15. Figure 2 placement. Figure 2 has moved from page 7 to page 3, directly after the introduction, which now references it (L. 52).

Reviewer_3Rz8OpenReview ↗

Authors' response assessment

Dear authors,

I thank you for your reply and clarifications. I found the answer of W.8 particularly important. Although the performance of the method is slightly below the baseline, its performance at inference on corrupted scenes is appealing. Unfortunately the tested augmentations are quite synthetic rather than physically plausible, it is unfortunate that they are not at least loosely explored, but authors clearly mention it for future works, which is reasonable.

The main issue that I still perceive in this manuscript is its synthetic feel. As mentioned in my initial review one of the main drawback of this work is not its research question, but rather its presentation. The manuscript is hard to read and while the rebuttal partly solved that issues many parts still feel tedious to go through. I acknowledge that the rebuttal substantially improved the clarity of the explanations, especially in the method section, but I believe it could be further clarified.

I am also still not fully convinced by the final performance of the method. I agree that it is effective at inference under corrupted settings, but it is still quite behind the actual baseline by a 2.68 in terms of mIoU, as indicated by table 5.

I would encourage authors to continue to simplify the reading experience of the manuscript until the end of the rebuttal.

I am strongly considering increasing my score, but I still need time to think and review the manuscript in more details.

I thank the authors for their answers and rebuttal.

AuthorsOpenReview ↗

Follow-up to Reviewer 3Rz8

Dear Reviewer,

Thank you for your thoughtful follow-up. We have uploaded an updated revision (readability edits marked in blue) and briefly address your remaining points below.

Readability. We further simplified dense passages to reduce notation overhead. Two examples can be checked directly in the main text: Sections 3.4 and 3.5 now state the two training-only objectives in plain terms (CORD scores fixed mixing ratios with an EMA teacher to guide the mean geometry weight; SA-TSI anchors swapped-point features to clean references and smooths shifts across same-class neighbors), moving five single-use graph and normalization symbols to Appendices B and C. Likewise, Table 5 replaces the checkmark grid with explicit row labels showing what each stage adds.

The 2.68 mIoU trade-off in Table 5. Comparing Row 2 (+ STS with fixed fusion, 71.97%) and Row 4 (full model, 69.29%), adding adaptive gating, CORD, and SA-TSI costs 2.68 points of clean mIoU in this setting. Note, however, that Row 2 actually degrades under shift relative to the unmodified baseline (lowering mean geometry-corruption mIoU from 47.25% to 43.18% and conflict Target Retention from 33.56% to 25.10%). Relative to Row 2, our full model raises mean geometry-corruption mIoU (which involves no test-time texture swaps) from 43.18% to 66.50% and Target Retention from 25.10% to 89.75%. Relative to the unmodified RGB+XYZ baseline (70.79%), the clean difference is -1.50 points on ScanNetV2-PointNeXt, whereas our full model improves clean mIoU in the other three dataset-backbone settings (+3.6, +2.1, and +0.4 points in Table 3).

Initial exploration of illumination and color shifts (Table 20). While real-sensor capture shifts remain future work, Appendix F.4 (Table 20, marked in green; summarized in Sections 4.4 and 4.5) provides an initial check using standard photometric perturbations (PhotoAug: brightness, contrast, saturation, hue, and grayscale shifts). Adding PhotoAug lifts our ScanNetV2-PointNeXt clean mIoU from 69.29% to 70.82% (within 0.64 points of Baseline + PhotoAug at 71.46%), while improving held-out external-texture mIoU from 46.33% to 59.80% and conflict Target Retention from 45.27% to 91.34%.

We appreciate your time and constructive feedback throughout the discussion.

Reviewer_ydmpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies RGB–geometry segmentation under misleading appearance. It combines geometry-preserving texture swaps and adaptive channel-wise fusion with candidate-relative routing distillation and graph-based residual invariance. The teacher and graph computations are used only during training.

Critical strengths
  • The motivation behind this work is interesting and novel to me.
  • The experiments evaluate affected-region segmentation, full-scene robustness to corruptions, and cross-domain transfer using two indoor datasets and two backbones.
Critical weaknesses
  • The strongest simple comparator is missing from the appearance-only full-scene tests. Geometry-only reaches 68.04 clean mIoU in Table 2. External texture projection and RGB-C leave geometry unchanged, so with RGB-independent preprocessing its predictions should be unchanged as well. This would potentially exceed the complete model's 55.40 and 63.88 in Table 5 and Table 6. Nevertheless, whether adaptive fusion improves the actual robustness–accuracy trade-off over discarding RGB is central to the method's practical value.
Questions for authors

Please refer to the weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ydmp

Thank you for reading our paper carefully and pointing out the missing comparison. Changes that address this comment are marked in orange in the revised PDF.

W1. Geometry-only comparison and the robustness–accuracy trade-off.

We have now evaluated geometry-only on the same shifted samples and added the results to Table 4 and Appendix F.3, Table 19.

Our framework performs better under geometry corruption and object-level texture conflicts, but geometry-only is stronger when the appearance of the whole scene is corrupted. On ScanNetV2 with PointNeXt, our method exceeds geometry-only by 16.14 mIoU points averaged over the three severity-3 geometry corruptions, by 16.18 conflict-region mIoU points under object-level texture conflicts, and by 1.25 mIoU points on clean scenes. These results show where keeping RGB and learning to fuse it with geometry helps.

As you anticipated, geometry-only performs better under the whole-scene RGB shifts and joint degradation we tested. We now make this trade-off clear in the paper. The comparison on ScanNetV2 with PointNeXt is below, with our result listed first and geometry-only second (Tables 2, 4, and 19):

Evaluation settingOursGeometry-onlyDifference
Clean69.2968.04+1.25
External texture55.4068.04−12.64
RGB-C63.8868.04−4.16
Geometry corruption66.5050.36+16.14
Joint degradation47.1750.36−3.19
Object-level texture conflict (affected-region mIoU)88.6472.46+16.18

The first five rows report full-scene mIoU (%), while the last row reports affected-region mIoU (%); Difference is Ours minus Geometry-only, in percentage points. Geometry corruption averages noise, jitter, and dropout at severity 3.

For object-level texture conflict (Table 2), TR is 89.75 vs. 82.37, and conflict-region mIoU is 88.64 vs. 72.46. These metrics are evaluated on the affected regions.

The model still uses RGB when geometry degrades. Replacing its RGB input with the training-set mean color lowers mIoU by 9.87 points under geometry corruption (Table 19, highlighted in green). Across the four evaluated shifts, geometry-only achieves higher mean/worst mIoU (59.20/50.36) than our method (58.24/47.17; Table 19). Section 4.4 and the conclusion now explicitly state this trade-off.

Photometric augmentation further improves this trade-off and complements our framework (Appendix F.4, Table 20, highlighted in green). It raises our clean mIoU from 69.29 to 70.82 and external-texture mIoU from 55.40 to 59.80. Under the same PhotoAug recipe, our framework exceeds the augmented baseline on external texture (59.80 vs. 46.33) and object-level conflict-region mIoU (90.69 vs. 26.59), with a 0.64-point clean-mIoU cost (70.82 vs. 71.46).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3Rz8 (Part 3 of 3): minor points and summary

W16. First sentence of the introduction. The opening sentence now focuses on indoor 3D semantic segmentation (L. 37–39).

W17. Suggested reference. Thank you. Kupyn & Rupprecht (2024) is now cited and discussed in the related work (L. 127–128).

W18. Table format. Both tables are resized and fit the text width. They are Table 12 and the former Table 19, which is now Table 21 because two tables were added.

W19. Argmin formatting. The index is now centered below arg min in Eq. 1, and in Eq. 14 of Appendix B.1.

W20. Indicator function. Yes, it is the indicator function. This is now stated (L. 179–180).

W21. Typo. Fixed.

W22. References. We corrected the three entries you identified: "Hongbo Fu" in the TransFusion entry, the author list of Ego-Exo4D, and the LaserMix page range (21705–21715). We also corrected the TransFusion page range (1090–1099).

W23. Paragraph formatting. Fixed. "Preference response" now starts a new paragraph (L. 476).

Summary of our answer to your overall assessment.

  • Clarity: we rewrote the abstract, revised the description of our method, and clarified the notation and terminology you highlighted (W1–W5, W9–W15).
  • Component contributions: adding SA-TSI improves all metrics in Table 5, including clean mIoU. The largest robustness gains in this cumulative ablation occur when GT-IGU and CORD are added together. Table 11 shows that simpler fusion also benefits from the augmentation, and that GT-IGU improves conflict robustness further (W7).
  • Metrics: the revised text distinguishes controlled conflict diagnostics from full-scene robustness evaluation. The latter shows improvements under appearance, geometry, and joint shifts beyond the training-time texture swap (W8).

Thank you again for the detailed suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.