acceptodds
Under review as a conference paper at ICLR 2027

Invalid Depth Is Not Missing Depth

Abstract

A missing depth measurement is not a measurement of zero. Yet common RGB-D unseen object instance segmentation pipelines encode invalid pixels as XYZ = (0, 0, 0), placing them at the optical centre. On OCID and OSD, 11–24% of pixels lack depth, whereas the synthetic training data are complete. Controlled comparisons and within-model ablations identify invalidity encoding as a substantial confound in robustness evaluation: UCN without validity filtering drops to zero Object-F after just 0.5% additional i.i.d. pixel removal, while the tested image-depth baseline remains stable. We further show that i.i.d. removal poorly matches the spatial structure of observed sensor failures, and introduce a corruption generator calibrated to each sensor's missing-depth statistics. Our repair, ProxFill, copies the XYZ triplet of the nearest valid pixel into an invalid pixel only when their image-space distance is at most τ. With τ fixed on a disjoint development split, this training-free rule improves the official UCN+ checkpoint by 2.8 Object-F and 6.9 Boundary-F points on OSD and transfers unchanged to MSMFormer. Under calibrated degradation, corrected UCN and MSMFormer remain within one Object-F point of their clean-input scores, while the published pipelines lose 14–16 points. Replacing the depth-validity filter with ProxFill also reduces measured UCN processing latency by 25%, without retraining or additional learnable parameters. These findings show that a substantial part of the measured robustness gap between RGB-D methods can arise from how invalid depth is represented.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
unseen object instance segmentation, RGB-D, epth sensing, invalid depth, input representation, robustness, benchmark evaluation, sensor noise modeling, instance segmentation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Cpz2OpenReview ↗
Rating
3(+1)
Confidence
1(−2)
Informativeness
2(−1)
Presentation
2(−1)
Soundness
2(−1)
Read review
Paper summary

This paper investigates a subtle but consequential issue in RGB-D unseen object instance segmentation: encoding invalid depth measurements as XYZ = (0, 0, 0) introduces misleading geometry and can significantly distort robustness evaluations. Through controlled experiments, the authors demonstrate the impact of invalid-depth encoding and show that independently removing depth pixels poorly represents real sensor failures. The paper further proposes a sensor-calibrated corruption protocol and ProxFill, a training-free method that fills invalid pixels using their nearest valid neighbors within a fixed distance threshold. Experiments on UCN, MSMFormer, and additional pipelines demonstrate substantial improvements in segmentation accuracy and robustness without modifying pretrained models.

Critical strengths

S1. A practically important and well-motivated diagnostic finding. The paper identifies an overlooked input-representation issue that substantially affects RGB-D segmentation performance. Improvements on naturally corrupted OSD inputs demonstrate that the finding is not merely an artifact of synthetic perturbations.

S2. Convincing mechanistic analysis. The inpainting ablation, invalid-point relocation experiments, and fixed-fill-mask shuffling controls provide strong evidence that local image–geometry correspondence matters. These experiments substantially strengthen the paper beyond a simple performance comparison.

S3. A useful reconsideration of robustness evaluation. The comparison between i.i.d. pixel removal and structured missing-depth patterns exposes limitations of conventional corruption testing. The proposed sensor-calibrated protocol provides a potentially valuable alternative for evaluating RGB-D robustness.

S4. Practical effectiveness and substantial experimental validation. ProxFill requires no retraining or additional learnable parameters and consistently improves multiple official checkpoints. Cross-backbone evaluations, held-out splits, sensitivity analyses, and inference-repeatability measurements further support the empirical findings.

Critical weaknesses

W1. Limited algorithmic novelty and incomplete comparisons with classical filling methods. ProxFill is essentially distance-gated nearest-neighbor interpolation. Although the paper evaluates unconditional filling and learned alternatives, direct comparisons with lightweight methods such as Telea interpolation or RGB-guided local filling under identical frozen checkpoints are missing. Such comparisons would better establish the specific value of the proposed rule.

W2. Limited validation of the sensor-calibrated corruption protocol. The corruption generator uses hole statistics and location priors estimated from the evaluated dataset, including unlabeled evaluation frames. While this is not label leakage, the protocol is not an entirely independent cross-sensor evaluation. Matching coarse hole statistics also does not establish realistic object-surface or material-dependent failure patterns. Consequently, the generality of the reported robustness gains under actual sensor degradation remains uncertain.

W3. Non-uniform generalization across datasets and resolutions. Appendix Table 11 reports a clean Object-F decrease from 84.01 to 83.16 for the full UOIS-Net-3D pipeline on OSD. On HouseCat6D at VGA resolution, Boundary-F decreases from 24.17 to 18.92 (Table 12). These results suggest that the fixed pixel-distance threshold may depend on image resolution, sensor characteristics, and downstream architectures. The conditions under which ProxFill is beneficial remain insufficiently characterized.

W4. The causal distinction between encoding and architecture is not fully established. Disabling inpainting within DA-Fusion rules out one confounding factor but does not isolate depth representation from architectural differences. Moreover, DA-Fusion still drops from 88.82 to 80.45 Object-F on OSD under +20% missing depth (Table 1). The evidence supports invalid-depth encoding as an important confound, but stronger claims about the relative robustness of image-depth and point-cloud architectures require qualification.

W5. Scene-level statistical dependence is not addressed. The paired bootstrap treats images as independent samples despite potential correlations within scenes or sequences. Although inference variability is carefully measured, scene-level confidence intervals would strengthen the smaller clean-input improvements, particularly on OCID.

Questions for authors

Q1. Regarding W1: Can the authors compare ProxFill against classical lightweight interpolation methods, such as Telea or RGB-guided local filling, using identical frozen checkpoints and evaluation settings?

Q2. Regarding W2: Do the robustness conclusions hold when the corruption shape bank and placement prior are constructed exclusively from scenes excluded from evaluation? Can the authors validate object-region missingness patterns rather than only aggregate hole statistics?

Q3. Regarding W3: What explains the Boundary-F degradation on HouseCat6D at VGA resolution and the clean-input degradation on UOIS-Net-3D? Does scaling the distance threshold with image resolution mitigate these failures?

Q4. Regarding W4: Given the remaining performance degradation of DA-Fusion under severe corruption, can the authors better separate the effects of input encoding from architectural differences, or narrow the corresponding claims?

Q5. Regarding W5: Do the clean-input improvements remain statistically significant under scene-level or sequence-level bootstrap resampling? How many independent scenes contribute to each evaluation split?

8 comments
AuthorsOpenReview ↗

Response to Reviewer Cpz2 (1/3)

Thank you for recognizing the diagnostic value and the controlled analysis. The requested experiments connect the contribution to three complementary findings: several classical local filling rules improve the same frozen model; gains persist with target data excluded from both generator components; and cluster-level intervals support robustness gains under additional missingness.

W1 / Q1. Classical filling baselines and novelty

The requested classical comparisons support the central diagnosis across several local filling rules. All methods use frozen UCN weights, all 111 OSD frames, three seeds, the original bank and retained filtering. Gated methods modify the same target pixels and preserve valid XYZ. Scores are Object-F / Boundary-F (%); +20% is requested removal of valid depth.

Frozen UCN inputClean+20% missingness
Zero XYZ86.41 / 67.4670.96 / 47.64
ProxFill89.12 / 74.5988.57 / 73.03
Unconditional nearest88.74 / 73.5887.01 / 69.26
Gated nearest depth + ray89.20 / 74.6688.68 / 73.21
Telea, all holes88.33 / 72.3486.44 / 67.29
Telea, same gate89.07 / 73.4288.95 / 72.37
RGB-guided, same gate89.28 / 75.1388.98 / 74.10

RGB-guided filling achieves the strongest mean scores in this comparison, while ProxFill provides a particularly simple training-free intervention. The gains shared by several local controls strengthen the diagnostic contribution: effective invalid-input handling improves the frozen model, and the result is not tied to one interpolation rule. Settings were fixed: Telea radius 3; RGB guidance uses up to 32 neighbors within 8 px, spatial standard deviation 4 px and color standard deviation 25/255. If its weight sum does not exceed 1e-30, it uses the ProxFill donor depth before target-ray reprojection.

ProxFill serves as a transparent reference intervention: it makes the input convention easy to test with an existing checkpoint.

W2 / Q2. Independent corruption sources and object patterns

The added experiment supports transfer when both generator components exclude the target collections. The bank and placement prior use only 120 ARID10 development frames, with τ=8 unchanged. This complements the original transductive protocol, whose statistics include unlabeled evaluation frames. Scores are Object-F / Boundary-F (%):

TargetAchieved added %Original + filterProxFill + filter
OCID held-out 20013.9189.31 / 76.2390.68 / 80.28
OSD 1119.7084.64 / 65.0589.03 / 73.82

OCID averages three seeds and OSD uses one. The achieved percentages are frame means; the frozen placement-attempt cap makes them lower than the requested 20%, so the transfer result applies to the reported doses. We also audited object-region missingness on all OCID200 frames. Native-invalid pixels have raw PNG label 0; the 188 frames with object labels have zero native object missingness and mean added object missingness of 2.2125%. The remaining 12 frames have no object labels and remain in the evaluation. These annotation-conditional measurements directly characterize object-region support, without treating it as material-dependent physical calibration.

Excluding target data from both shape and placement statistics tests the complete generator. Gains on both target collections extend the evidence beyond the original target-derived protocol at the reported achieved doses.

AuthorsOpenReview ↗

Response to Reviewer Cpz2 (2/3)

W3 / Q3. Operating conditions and resolution scaling

The added controls identify useful operating conditions and show that scale-aware gating mitigates the VGA boundary effect. On the fixed HouseCat6D pilot (100 frames, five scenes, one seed), native ProxFill improves Object-F / Boundary-F from 41.72 / 21.03 to 54.72 / 21.40. At VGA, scaling the native 8-px gate to 4.59 px raises Boundary-F from 18.94 to 22.25; the zero-input reference is 24.29. All five VGA scene means improve over the fixed gate. The reverse scaling to 13.95 px at native resolution changes Boundary-F from 21.40 to 19.05. Both scaling anchors and all scenes are retained; depth/labels use exact nearest-neighbor resizing, color uses area interpolation, and intrinsics are adjusted.

The matched target-ray, Telea and RGB-guided controls also retain the VGA boundary loss, including when filtering is disabled. This helps distinguish the observed behavior from a problem specific to complete XYZ copying or the validity filter. The controls establish partial mitigation and condition dependence; resizing also changes predictor features, so they do not isolate a unique cause.

For UOIS, the useful robustness effect remains substantial: full-pipeline Object-F improves 70.00→82.97 at requested +20% missingness. Clean Object-F is 84.01→83.16, and processed initial masks are 83.32→82.67. Later stages also read variant depth, so an RGB-only explanation is not established. We therefore characterize the benefit by input condition and downstream pipeline, with the strong additional-missingness improvement reported alongside the clean-input change.

W4 / Q4. Encoding versus architecture

The most direct evidence comes from interventions within the same frozen model. In the archived UCN control on all 111 OSD frames, the τ=8 fill set and complete XYZ multiset are fixed, and only spatial assignment changes. With filtering off, local versus shuffled Object-F is 88.98 versus 84.91 clean and 88.51 versus 54.60 at requested +20%. The three inference repeats reuse the same corruption and permutation. This supports the practical importance of spatial input handling within UCN. DA-Fusion's remaining 8.37-point drop is fully compatible with that conclusion: the claim concerns an input-handling confound, while cross-model comparisons also involve architecture and training. They do not establish an encoding-only ranking or immunity to missing depth.

AuthorsOpenReview ↗

Response to Reviewer Cpz2 (3/3)

W5 / Q5. Scene dependence and statistical significance

The cluster-level analysis supports positive gains under additional missingness, while quantifying the uncertainty of the smaller clean-input changes. OCID1570 uses 50 directory clusters, merging paired top/bottom views (26 ARID20 and 24 YCB10), with 20,000 paired whole-cluster draws. The new OCID200 analysis averages each frame over three seeds and uses the same clusters. The following are gains in percentage points with descriptive 95% intervals:

Condition / metricGain95% cluster interval
OCID1570 clean Object-F, frame-weighted+0.317[0.009, 0.653]
OCID1570 clean Object-F, equal-cluster+0.266[-0.036, 0.603]
YCB10 clean Object-F+0.124[-0.305, 0.639]
OCID1570 corrupted Object-F+3.174[2.527, 3.872]
OCID1570 corrupted Boundary-F+8.876[7.797, 10.064]
OCID200 clean Object-F+0.330[-0.086, 0.784]
OCID200 corrupted Object-F+1.374[0.703, 2.041]

Whole-cluster resampling keeps grouped frames together; averaging OCID200 seeds before resampling also prevents repeated runs from being counted as additional scene observations.

The OCID200 corrupted result uses the independent-source experiment at achieved 13.91% added removal. Its post-hoc leave-one-cluster-out Object-F gains remain +1.225 to +1.565; these are point estimates, and the primary result retains all 50 clusters. Some clean intervals include zero, so the strongest statistical support concerns additional-missingness robustness. The intervals have no multiplicity correction, directory clusters are not proof of physical independence, and OSD does not provide verified scene IDs.

The combined evidence supports the core contribution: controlled diagnosis of an input convention, a simple intervention, and robustness evaluation informed by observed missing-depth patterns.

Reviewer_dTvkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates invalid-depth encoding in RGB-D unseen object instance segmentation. Encoding missing depth as zero coordinates introduces artificial geometry into point clouds, potentially affecting object separation and robustness evaluation. Through controlled experiments with frozen model weights, the authors analyze this issue, introduce a corruption protocol based on observed missing-region shapes, and propose ProxFill, which copies nearby valid XYZ coordinates into invalid pixels only within a distance threshold. Experiments show improved segmentation without retraining and reduced processing latency when replacing the original depth-validity filtering stage.

Critical strengths

A clearly identified problem with practical relevance. The paper highlights an easily overlooked input-processing issue and explains how it can affect model comparisons and robustness conclusions. This is useful for reproducibility and evaluation.

Well-targeted analysis of the underlying mechanism. Controls involving disabled depth inpainting, relocated invalid points, and shuffled fill values help distinguish the effects of encoding, filling, and local geometric consistency.

An insightful discussion of robustness evaluation. The paper demonstrates substantial spatial differences between independent pixel removal and observed missing-depth regions, showing that corruption design can materially change robustness assessments.

A simple method with reasonably comprehensive validation. ProxFill requires no training and is evaluated across different backbones and datasets. The paper also discusses stochastic variation, threshold sensitivity, and limitations, providing useful boundaries for its claims.

Critical weaknesses

The technical novelty lies primarily in diagnosis and validation. ProxFill itself is a straightforward distance-gated nearest-neighbor filling rule. The paper should more clearly distinguish its methodological contribution from its diagnostic and evaluation contributions.

The attribution of gains in the main results needs clarification. The final pipeline both introduces ProxFill and removes the original depth-validity filter. Although relevant controls are referenced, the main results do not clearly separate the accuracy and efficiency contributions of these changes.

The validation of “sensor calibration” is limited. The generator primarily matches missing-region shapes, spatial distributions, and connected-component statistics. These statistics do not establish whether it preserves relationships between missing depth and object materials, depth boundaries, or scene geometry. Improvements under synthetic corruption therefore require cautious interpretation as evidence of robustness to real sensor degradation.

The applicability of the threshold across imaging settings remains unclear. The method uses image-space distance, whose corresponding physical extent varies with resolution, camera parameters, and object distance. The cross-dataset results support some transferability but do not establish stability across broader imaging conditions.

The robustness conclusions should be more narrowly scoped. The evidence mainly concerns existing pipelines using a particular point-cloud input convention. It does not establish that the findings extend to methods that explicitly handle invalid depth or use other geometric representations. Some broader statements about robustness gaps should be qualified accordingly.

Questions for authors

Based on the existing ablations, could the authors separately summarize the effects of ProxFill, removing the depth-validity filter, and combining both changes? This would clarify the sources of the accuracy gains and latency reduction.

Which data splits were used to construct the missing-region shape bank and spatial distribution? Did they include final evaluation images or images from the same acquisition sequences? Please clarify the data-access assumptions of the protocol.

Beyond component counts, areas, and boundary concentration, does any existing analysis examine the relationship between synthetic missing regions, object regions, and depth boundaries? If not, would “a corruption protocol based on observed missing-depth statistics” more accurately describe the scope of the calibration?

Were input resolutions and preprocessing consistent across datasets in the threshold-transfer experiments? Do the existing threshold sweeps support retaining a fixed pixel-distance threshold when the imaging scale changes?

ProxFill copies a neighboring pixel’s complete XYZ coordinates into the target pixel. Could the authors explain its effect on local geometry and why this was chosen over propagating depth and reprojecting it along the target pixel’s camera ray?

3 comments
AuthorsOpenReview ↗

Response to Reviewer dTvk (1/3)

Thank you for recognizing the diagnostic contribution and the controlled analysis. Our response distinguishes three contributions: identifying an input-handling confound, testing it with a simple training-free intervention, and evaluating robustness under observed missing-depth patterns. The added controls isolate filling from filtering, remove target contributions from both generator components, and quantify transfer across imaging scales.

W1. Contribution and novelty

The contribution is a controlled diagnosis of a consequential input convention and its effect on robustness evaluation. ProxFill provides a simple, training-free intervention with which to test that diagnosis. In the archived matched control, we fix the τ=8 filled-pixel set and preserve the complete XYZ multiset, changing only its spatial assignment. With filtering disabled on all 111 OSD frames, local versus shuffled XYZ gives Object-F 88.98 versus 84.91 clean and 88.51 versus 54.60 at requested +20% missingness. Thus filling coverage and the point multiset alone do not explain the observed benefit; local image–geometry correspondence matters. The three inference repeats reuse the same corruption and permutation, and the contrast is a strong nonlocal perturbation rather than a comparison of local interpolation optimality.

The training-free intervention makes this diagnosis directly testable with an existing checkpoint. Its simplicity keeps the role of input handling explicit.

W2 / Q1. Filling, filtering and latency

The accuracy contribution of filling remains clear when the original validity filter is retained. The following matched comparison uses frozen UCN, all 111 OSD frames, three seeds and the original bank. Each on/off comparison reuses the same raw prediction. Scores are Object-F / Boundary-F (%); +20% is requested removal of valid depth.

Input / filterClean+20% missingness
Zero / on86.41 / 67.4670.96 / 47.64
Zero / off82.00 / 58.0370.51 / 45.95
ProxFill / on89.12 / 74.5988.57 / 73.03
ProxFill / off89.10 / 74.5488.47 / 72.68

Retaining the original filter therefore suffices to realize the segmentation gain; removing it is a separate pipeline choice.

The fixed-reference control further separates prediction changes from filtering support. With pre-fill support fixed, changing predictions improves Object-F by +2.915 clean and +5.747 corrupted (achieved 20.13%). Changing support alone on fixed ProxFill predictions gives -0.210 and +11.862 points. These controls establish a prediction benefit even without changing support; their interaction means the total cannot be assigned a unique percentage decomposition.

For efficiency, the archived A800 result is an implementation-specific stage-sum reduction of 84.73→63.15 ms (25.5%). On the separate server (20 clean OSD frames, three repeats), measured totals for zero/on, zero/off, fill/on and fill/off are 189.77, 182.74, 196.51 and 189.54 ms. Combined-minus-baseline is -0.232 ms with descriptive 95% interval [-1.738, 1.044]. The new total-latency interval includes zero. We report the demonstrated accuracy benefit and the archived stage-sum reduction separately from these hardware-dependent totals, which exclude file I/O, scoring and crop refinement.

AuthorsOpenReview ↗

Response to Reviewer dTvk (2/3)

W3 / Q2. Data used by the shape bank and placement prior

The added source-transfer experiment tests whether gains persist without target contributions to the generator: both the shape bank and placement prior are built solely from 120 ARID10 development frames, excluding the target collections, with τ=8 unchanged. The original protocol uses unlabeled evaluation frames in both statistics and is transductive. The new experiment retains improvements after removing target contributions from both components:

TargetAchieved added %Original + filterProxFill + filter
OCID held-out 20013.9189.31 / 76.2390.68 / 80.28
OSD 1119.7084.64 / 65.0589.03 / 73.82

Scores are Object-F / Boundary-F (%). OCID averages three seeds and OSD uses one. The achieved percentages are frame means: both experiments request 20%, while the frozen placement-attempt cap limits the attained dose. These results support transfer at the reported doses; OSD filename separation is not treated as verified acquisition-scene independence.

This tests source transfer for the complete generator while retaining the selected distance gate.

W3 / Q3. Object regions, boundaries and calibration

The additional region analysis directly measures object, boundary and observable depth-edge concentration. For each frame, we divide regional missingness by whole-image missingness and then average the ratios. Native/added ratios are 0.114/0.627 inside objects, 0.561/0.582 near boundaries and 0.341/2.012 at observable depth edges. Thus the analysis characterizes both the similar boundary concentration and the distinct object/depth-edge distributions. These are annotation-conditional spatial statistics from the original OSD protocol. The contribution is a corruption protocol based on observed missing-depth statistics. These spatial measurements characterize the protocol; they do not model material-dependent sensor failures.

W4 / Q4. Resolution and threshold transfer

The controlled scale experiment gives a practical answer: adjusting the gate can recover part of the VGA boundary loss. OCID/OSD use 640×480; HouseCat6D compares native 1096×852 with VGA on the same 100 frames from five scenes. Color uses area interpolation, depth/labels use exact nearest-neighbor resizing, and intrinsics are adjusted. Each scaling anchor uses the square root of the target/reference pixel-count ratio. With filtering retained, the complete two-anchor comparison is:

InputZero inputFixed 8 pxNative-8 scalingVGA-8 scaling
VGA51.16 / 24.2951.59 / 18.9455.93 / 22.2551.59 / 18.94
Native41.72 / 21.0354.72 / 21.4054.72 / 21.4053.34 / 19.05

Scores are Object-F / Boundary-F (%). Native-8 gives 4.59 px at VGA; VGA-8 gives 13.95 px at native resolution. Scaling improves VGA Boundary-F from 18.94 to 22.25, with the zero-input value 24.29 providing the reference. This quantifies useful mitigation without treating the pixel gate as scale-invariant; resizing also changes predictor features.

AuthorsOpenReview ↗

Response to Reviewer dTvk (3/3)

Q5. XYZ copying versus target-ray reprojection

Full-vector copying is a minimal local intervention: it preserves a measured donor triplet without another backprojection. The target-ray control is informative because it tests whether the segmentation benefit survives a different geometrically local treatment. With the same gate and original OSD bank, target-ray gives 89.20 / 74.66 clean and 88.68 / 73.21 corrupted, compared with XYZ copying at 89.12 / 74.59 and 88.57 / 73.03. These similar mean scores support local input treatment as the central factor, rather than a unique advantage of copying the full vector.

Geometrically, copied XYZ minus target-ray XYZ is (z Δu/f_x, z Δv/f_y, 0) for a zero-skew pinhole model, with offsets from target to donor. A post-hoc OSD check on 111 frames at requested +20% gives frame-mean XYZ discrepancies of 16.86 mm for copying and 12.44 mm for target-ray; both have 11.26 mm axial discrepancy. This uses synthetically removed, gated pixels and original recorded measurements as reference, not physical ground truth; native holes have no reference. The comparison distinguishes segmentation-input utility from physical surface reconstruction.

W5. Scope of the robustness conclusions

The supported conclusion is a within-pipeline one: changing invalid-input handling can materially change accuracy and measured robustness while weights remain fixed. This makes input conventions an important consideration for reproducible model comparison. Cross-model rankings also involve architecture and training, so they do not isolate an encoding-only effect; models with explicit validity handling should be evaluated in their own setting.

Together, these controls show that input handling is a consequential, testable part of evaluating frozen segmentation systems.

Reviewer_YrE9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces ProxFill, a method that aims at repairing missing observations in the RGB-D sensors. The mechanism advantage is that it is training-free and completes point-cloud-encoded observations, enabling better segmentation. The validation is performed on two selected datasets, showing an improved performance.

Critical strengths
  • The paper rightly observes an important gap in research: Taking a closer look on the sensor encodings, rather than deploying inpainting models or other post-processing techniques to infer missing observations. As claimed here, this can be done training-free.
Critical weaknesses
  • Yet, the paper is somewhat naively approaching the issue. There is no study presented on the actual sensor acquisition principle and the actual encoding. Also the experiments are limited in terms of the used RGB-D sensors. Is it specific to a sensor? Generalisable to other sensors? The paper would benefit from a discussion in the method assumption and in the experiments. Albeit the central Figure 3 argues on the analysis of the signal, there is no present. The “copying” of the recorded XYZ values within a defined threshold surrounding, does not engage with the sensing principle (it is operating on the already recorded signal).
  • The previous point links up here: Indeed the paragraph starting at Line 319 invites questions (Tab 16 as well). The tested performance on ARID and OSD provides a mixed picture. The method seems to be quite sensitive to hyperparameters chosen, the performance varies quite a lot among the two datasets already. What would happen across the other experimental data? As above, there is no analysis to the sensor’s used, and it is hard to infer whether this can scale up to other RGB-D sensors and setups. RGB-D can be: stereo vision, structured light, ToF, etc. Them all have different acquisition geometry and behave differently in outdoor, indoor, mixed light scenarios. This has not been investigated.
Questions for authors
  • I appreciate the simplicity of the method. However, the simplicity also requires some stronger justifications. Of course, please also clarify if there is any misunderstanding on my end.
  • I would like to ask the authors to discuss (quantitatively and qualitatively) that the presented naive approach scales up to a) other scenarios conditions, and b) other sensors (a few selected baselines and data)
  • I would also like to ask to elaborate on the physics behind the sensor that are assumed here. It is hard to gather from the paper on what conceptual principles the ProxFill averaging with a single hyperparameter should work (even though indeed we can observe an increase on a selected examples)
  • Additionally, the paper argues for point cloud encoding – this implies that this will be more beneficial for sensors relying on, e.g., ToF(?) In general, then, what about LiDAR then, and the angular resolution of it?
11 comments
AuthorsOpenReview ↗

Response to Reviewer YrE9 (1/2)

Thank you for recognizing the importance of examining sensor encodings and the simplicity of a training-free intervention. We address sensor transfer with an additional active-stereo dataset, connect the distance gate to camera geometry, and distinguish measured RGB-D results from the analytical extension to LiDAR. Below, W1 and W2 refer to the two Critical Weaknesses paragraphs, addressed together with your questions.

W1, W2 / Q1. Sensor and imaging-scale transfer

The additional HouseCat6D experiment extends the evidence to a different indoor sensing setup and shows a substantial native-resolution Object-F improvement. OCID uses ASUS Xtion cameras, whereas HouseCat6D uses D435/D435i active stereo. We evaluate the same 100 frames from five HouseCat6D scenes, with frozen UCN, one seed and validity filtering retained. Scores are Object-F / Boundary-F (%); all filling methods below use the same 8-px target mask at each resolution.

InputNative 1096×852VGA 640×480
Zero input41.72 / 21.0351.16 / 24.29
ProxFill54.72 / 21.4051.59 / 18.94
Target-ray54.57 / 21.4351.71 / 18.96
Gated Telea55.01 / 21.6951.30 / 18.56
Gated RGB-guided55.48 / 22.3151.19 / 18.97

At native resolution, all four local filling rules improve both mean F scores over zero input. This supports the broader input-handling diagnosis across the tested indoor sensor setups. The VGA boundary results also identify imaging scale as an important deployment condition; the controlled scaling experiment below directly examines this behavior.

With the checkpoint and filtering fixed, these comparisons directly test how input preparation transfers to the new sensing setup.

AuthorsOpenReview ↗

Response to Reviewer YrE9 (2/2)

W1, W2 / Q2. Physical assumption and threshold choice

ProxFill uses local continuity of observed geometry to construct a useful segmentation input. It copies one valid neighbor's complete XYZ triplet; it does not average measurements. The intervention is applied after acquisition. Xtion uses projected-pattern triangulation, while D435 estimates stereo disparity with projected texture aiding correspondence. These different acquisition mechanisms motivate studying invalid-input handling at the representation level.

The distance gate restricts copying to nearby observations. Under a pinhole model, its nominal physical extent scales approximately as zτ/f, connecting the pixel threshold to depth and camera calibration. For the HouseCat6D scale control, we adjust intrinsics and use area interpolation for color and exact nearest-neighbor resizing for depth and labels. Scaling native τ=8 to 4.59 px at VGA raises Boundary-F from 18.94 to 22.25, compared with 24.29 for zero input. Conversely, scaling VGA-8 to 13.95 px at native resolution changes Boundary-F from 21.40 to 19.05. Together, these controls explain why image scale must be considered when transferring a pixel gate; the VGA recovery is partial, and resizing also changes network features.

The original 120-frame, six-candidate ARID10 search selected τ=8. In the later 200-frame, 15-radius diagnostic, the best mean clean/corrupted Object-F at radius 10 is only 0.097 points higher. OSD’s separate clean/corrupted maxima occur at radii 16/6. These sweeps characterize a useful operating range at the tested scale, with condition-dependent optima; the transfer comparisons retain the selected gate. Local continuity is an input assumption rather than a guarantee of surface reconstruction, particularly at occlusions, depth discontinuities or large holes.

W1 / Q3. Point clouds, ToF and LiDAR

The relevant property is how acquired measurements are represented and handled by the downstream model. Organized XYZ is not specific to ToF. Our empirical evidence concerns indoor RGB-D; for ToF and LiDAR, the following is an analytical extension. For common-origin rays at equal radial range r and angular separation θ, point spacing is 2r sin(θ/2), approximately rθ for small θ in radians. A suitable LiDAR neighborhood therefore needs to account for beam calibration and range. This provides the geometric connection to angular resolution without assuming that a fixed image-pixel gate transfers unchanged.

All reported comparisons retain the frozen model weights. The additional controls clarify where the simple intervention is useful and connect its behavior to sensor representation and imaging scale.

Thank you for your answers and engaging in the discussion.

The extended experiment on the HouseCat6D partially addresses the comment. However, it is not a new setting but rather an extension of the previously provided setup (e.g., same lighthing conditions, camera-object distance) - how would the method behave in the outdoor setting, for example? Could you provide more insights? Again, it is not only about the performance per se, but rather the working principle operating on the already recorded data without engaging much with the physics of the sensor. Also, the reported results are oscillating in a rather modest improvement - the improvement is not "substantial". I would tone down this claim and rather argue for a robust or consistent results.

Looking forward to hearing more about the method!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated Manuscript and Summary of Revisions

We thank all reviewers and the Area Chair for their time, careful reading, and constructive feedback. We particularly appreciate the reviewers' willingness to engage in a detailed discussion of the method and its supporting evidence. Following our rebuttal, we continued revising the manuscript and have now uploaded the updated version, incorporating the additional experiments and clarifications discussed in our responses.

For ease of review, we summarize the main additions below. All section, table, and page references refer to the revised manuscript.

  1. Controlled attribution of the gains. The filling/filtering controls and fixed-reference analysis show that ProxFill improves predictions with the original support reference held fixed: +2.915 Object-F points on clean input and +5.747 under corruption. This strengthens the evidence for the contribution of input handling within the same frozen model. See Section 6.1 and Table 6 (p. 10), with matched classical alternatives and implementation details in Appendix N.

  2. Independent corruption sources and statistical support. Gains persist when both the corruption shape bank and spatial prior are constructed exclusively from the development split, without using statistics from the target collections. The revision also incorporates regional missingness diagnostics and whole-cluster uncertainty estimates for the gains under corruption. These complementary controls strengthen the evaluation of robustness. See Section 6.1 and Appendix N.

  3. Working principle and cumulative corruption. We have added a conditional analysis of local support and tests that progressively enlarge missing regions and perturb the remaining measurements. Across all 111 OSD frames, ProxFill achieves 88.69 Object-F in the strongest tested condition combining missingness and measurement noise, compared with 75.67 for the baseline that sets missing points to zero. The original gate, checkpoint, and filter are retained. See Sections 5 and 6.1 and Appendices O–P.

  4. Sensor transfer and calibration. On real ToF data comprising 52 frames from 13 scenes, the original gate, frozen checkpoint, and retained filter yield aggregate gains of +3.00–3.70 Object-F points and +1.39–2.56 Boundary-F points across five inference seeds. We also incorporate HouseCat6D imaging-scale controls and a calibrated outdoor LiDAR test of geometric locality. The protocols, scene-level results, and coverage analyses appear in Appendix N and Appendices Q–R; Section 6.1 summarizes the main findings.

Together, these additions reinforce ProxFill's contribution: a simple input intervention that improves segmentation robustness within the evaluated frozen pipelines, without retraining. The original structure and core results are retained, with key additions in the 10-page main text and detailed analyses in the appendices. We hope this consolidated evidence will support a favorable assessment of the paper's contribution, and we welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.