- Rating
- 3(+1)
- Confidence
- 1(−2)
- Informativeness
- 2(−1)
- Presentation
- 2(−1)
- Soundness
- 2(−1)
Read review
Paper summary
This paper investigates a subtle but consequential issue in RGB-D unseen object instance segmentation: encoding invalid depth measurements as XYZ = (0, 0, 0) introduces misleading geometry and can significantly distort robustness evaluations. Through controlled experiments, the authors demonstrate the impact of invalid-depth encoding and show that independently removing depth pixels poorly represents real sensor failures. The paper further proposes a sensor-calibrated corruption protocol and ProxFill, a training-free method that fills invalid pixels using their nearest valid neighbors within a fixed distance threshold. Experiments on UCN, MSMFormer, and additional pipelines demonstrate substantial improvements in segmentation accuracy and robustness without modifying pretrained models.
Critical strengths
S1. A practically important and well-motivated diagnostic finding. The paper identifies an overlooked input-representation issue that substantially affects RGB-D segmentation performance. Improvements on naturally corrupted OSD inputs demonstrate that the finding is not merely an artifact of synthetic perturbations.
S2. Convincing mechanistic analysis. The inpainting ablation, invalid-point relocation experiments, and fixed-fill-mask shuffling controls provide strong evidence that local image–geometry correspondence matters. These experiments substantially strengthen the paper beyond a simple performance comparison.
S3. A useful reconsideration of robustness evaluation. The comparison between i.i.d. pixel removal and structured missing-depth patterns exposes limitations of conventional corruption testing. The proposed sensor-calibrated protocol provides a potentially valuable alternative for evaluating RGB-D robustness.
S4. Practical effectiveness and substantial experimental validation. ProxFill requires no retraining or additional learnable parameters and consistently improves multiple official checkpoints. Cross-backbone evaluations, held-out splits, sensitivity analyses, and inference-repeatability measurements further support the empirical findings.
Critical weaknesses
W1. Limited algorithmic novelty and incomplete comparisons with classical filling methods. ProxFill is essentially distance-gated nearest-neighbor interpolation. Although the paper evaluates unconditional filling and learned alternatives, direct comparisons with lightweight methods such as Telea interpolation or RGB-guided local filling under identical frozen checkpoints are missing. Such comparisons would better establish the specific value of the proposed rule.
W2. Limited validation of the sensor-calibrated corruption protocol. The corruption generator uses hole statistics and location priors estimated from the evaluated dataset, including unlabeled evaluation frames. While this is not label leakage, the protocol is not an entirely independent cross-sensor evaluation. Matching coarse hole statistics also does not establish realistic object-surface or material-dependent failure patterns. Consequently, the generality of the reported robustness gains under actual sensor degradation remains uncertain.
W3. Non-uniform generalization across datasets and resolutions. Appendix Table 11 reports a clean Object-F decrease from 84.01 to 83.16 for the full UOIS-Net-3D pipeline on OSD. On HouseCat6D at VGA resolution, Boundary-F decreases from 24.17 to 18.92 (Table 12). These results suggest that the fixed pixel-distance threshold may depend on image resolution, sensor characteristics, and downstream architectures. The conditions under which ProxFill is beneficial remain insufficiently characterized.
W4. The causal distinction between encoding and architecture is not fully established. Disabling inpainting within DA-Fusion rules out one confounding factor but does not isolate depth representation from architectural differences. Moreover, DA-Fusion still drops from 88.82 to 80.45 Object-F on OSD under +20% missing depth (Table 1). The evidence supports invalid-depth encoding as an important confound, but stronger claims about the relative robustness of image-depth and point-cloud architectures require qualification.
W5. Scene-level statistical dependence is not addressed. The paired bootstrap treats images as independent samples despite potential correlations within scenes or sequences. Although inference variability is carefully measured, scene-level confidence intervals would strengthen the smaller clean-input improvements, particularly on OCID.
Questions for authors
Q1. Regarding W1: Can the authors compare ProxFill against classical lightweight interpolation methods, such as Telea or RGB-guided local filling, using identical frozen checkpoints and evaluation settings?
Q2. Regarding W2: Do the robustness conclusions hold when the corruption shape bank and placement prior are constructed exclusively from scenes excluded from evaluation? Can the authors validate object-region missingness patterns rather than only aggregate hole statistics?
Q3. Regarding W3: What explains the Boundary-F degradation on HouseCat6D at VGA resolution and the clean-input degradation on UOIS-Net-3D? Does scaling the distance threshold with image resolution mitigate these failures?
Q4. Regarding W4: Given the remaining performance degradation of DA-Fusion under severe corruption, can the authors better separate the effects of input encoding from architectural differences, or narrow the corresponding claims?
Q5. Regarding W5: Do the clean-input improvements remain statistically significant under scene-level or sequence-level bootstrap resampling? How many independent scenes contribute to each evaluation split?
8 comments
Response to Reviewer Cpz2 (2/3)
W3 / Q3. Operating conditions and resolution scaling
The added controls identify useful operating conditions and show that scale-aware gating mitigates the VGA boundary effect. On the fixed HouseCat6D pilot (100 frames, five scenes, one seed), native ProxFill improves Object-F / Boundary-F from 41.72 / 21.03 to 54.72 / 21.40. At VGA, scaling the native 8-px gate to 4.59 px raises Boundary-F from 18.94 to 22.25; the zero-input reference is 24.29. All five VGA scene means improve over the fixed gate. The reverse scaling to 13.95 px at native resolution changes Boundary-F from 21.40 to 19.05. Both scaling anchors and all scenes are retained; depth/labels use exact nearest-neighbor resizing, color uses area interpolation, and intrinsics are adjusted.
The matched target-ray, Telea and RGB-guided controls also retain the VGA boundary loss, including when filtering is disabled. This helps distinguish the observed behavior from a problem specific to complete XYZ copying or the validity filter. The controls establish partial mitigation and condition dependence; resizing also changes predictor features, so they do not isolate a unique cause.
For UOIS, the useful robustness effect remains substantial: full-pipeline Object-F improves 70.00→82.97 at requested +20% missingness. Clean Object-F is 84.01→83.16, and processed initial masks are 83.32→82.67. Later stages also read variant depth, so an RGB-only explanation is not established. We therefore characterize the benefit by input condition and downstream pipeline, with the strong additional-missingness improvement reported alongside the clean-input change.
W4 / Q4. Encoding versus architecture
The most direct evidence comes from interventions within the same frozen model. In the archived UCN control on all 111 OSD frames, the τ=8 fill set and complete XYZ multiset are fixed, and only spatial assignment changes. With filtering off, local versus shuffled Object-F is 88.98 versus 84.91 clean and 88.51 versus 54.60 at requested +20%. The three inference repeats reuse the same corruption and permutation. This supports the practical importance of spatial input handling within UCN. DA-Fusion's remaining 8.37-point drop is fully compatible with that conclusion: the claim concerns an input-handling confound, while cross-model comparisons also involve architecture and training. They do not establish an encoding-only ranking or immunity to missing depth.
Response to Reviewer Cpz2 (3/3)
W5 / Q5. Scene dependence and statistical significance
The cluster-level analysis supports positive gains under additional missingness, while quantifying the uncertainty of the smaller clean-input changes. OCID1570 uses 50 directory clusters, merging paired top/bottom views (26 ARID20 and 24 YCB10), with 20,000 paired whole-cluster draws. The new OCID200 analysis averages each frame over three seeds and uses the same clusters. The following are gains in percentage points with descriptive 95% intervals:
| Condition / metric | Gain | 95% cluster interval |
|---|---|---|
| OCID1570 clean Object-F, frame-weighted | +0.317 | [0.009, 0.653] |
| OCID1570 clean Object-F, equal-cluster | +0.266 | [-0.036, 0.603] |
| YCB10 clean Object-F | +0.124 | [-0.305, 0.639] |
| OCID1570 corrupted Object-F | +3.174 | [2.527, 3.872] |
| OCID1570 corrupted Boundary-F | +8.876 | [7.797, 10.064] |
| OCID200 clean Object-F | +0.330 | [-0.086, 0.784] |
| OCID200 corrupted Object-F | +1.374 | [0.703, 2.041] |
Whole-cluster resampling keeps grouped frames together; averaging OCID200 seeds before resampling also prevents repeated runs from being counted as additional scene observations.
The OCID200 corrupted result uses the independent-source experiment at achieved 13.91% added removal. Its post-hoc leave-one-cluster-out Object-F gains remain +1.225 to +1.565; these are point estimates, and the primary result retains all 50 clusters. Some clean intervals include zero, so the strongest statistical support concerns additional-missingness robustness. The intervals have no multiplicity correction, directory clusters are not proof of physical independence, and OSD does not provide verified scene IDs.
The combined evidence supports the core contribution: controlled diagnosis of an input convention, a simple intervention, and robustness evaluation informed by observed missing-depth patterns.
Response to Reviewer Cpz2 (1/3)
Thank you for recognizing the diagnostic value and the controlled analysis. The requested experiments connect the contribution to three complementary findings: several classical local filling rules improve the same frozen model; gains persist with target data excluded from both generator components; and cluster-level intervals support robustness gains under additional missingness.
W1 / Q1. Classical filling baselines and novelty
The requested classical comparisons support the central diagnosis across several local filling rules. All methods use frozen UCN weights, all 111 OSD frames, three seeds, the original bank and retained filtering. Gated methods modify the same target pixels and preserve valid XYZ. Scores are Object-F / Boundary-F (%); +20% is requested removal of valid depth.
RGB-guided filling achieves the strongest mean scores in this comparison, while ProxFill provides a particularly simple training-free intervention. The gains shared by several local controls strengthen the diagnostic contribution: effective invalid-input handling improves the frozen model, and the result is not tied to one interpolation rule. Settings were fixed: Telea radius 3; RGB guidance uses up to 32 neighbors within 8 px, spatial standard deviation 4 px and color standard deviation 25/255. If its weight sum does not exceed 1e-30, it uses the ProxFill donor depth before target-ray reprojection.
ProxFill serves as a transparent reference intervention: it makes the input convention easy to test with an existing checkpoint.
W2 / Q2. Independent corruption sources and object patterns
The added experiment supports transfer when both generator components exclude the target collections. The bank and placement prior use only 120 ARID10 development frames, with τ=8 unchanged. This complements the original transductive protocol, whose statistics include unlabeled evaluation frames. Scores are Object-F / Boundary-F (%):
OCID averages three seeds and OSD uses one. The achieved percentages are frame means; the frozen placement-attempt cap makes them lower than the requested 20%, so the transfer result applies to the reported doses. We also audited object-region missingness on all OCID200 frames. Native-invalid pixels have raw PNG label 0; the 188 frames with object labels have zero native object missingness and mean added object missingness of 2.2125%. The remaining 12 frames have no object labels and remain in the evaluation. These annotation-conditional measurements directly characterize object-region support, without treating it as material-dependent physical calibration.
Excluding target data from both shape and placement statistics tests the complete generator. Gains on both target collections extend the evidence beyond the original target-derived protocol at the reported achieved doses.