- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This submission studies how to fuse language-conditioned responses across views when a projected 3D surface may be occluded. The main observation is that partial occlusion can retain useful semantic evidence through spatially extended response maps, whereas fully occluded responses carry much weaker membership information. The proposed fusion rule combine class-conditional visible, and partially occluded response densities with a shared full-occlusion density, using estimated visibility and a local spatial gate that avoid instance-grouping labels at inference. The work also introduces VisCen3D for surface-level visibility diagnosis. On fresh synthetic scenes with analytic surface tracks, the method improves occluded-surface ranking and target-track selection over development-selected baselines. The real-data and estimated-geometry experiments are less favorable because the gains depend strongly on the response readout, and incomplete surface recovery substantially limits the practical benefit.
Critical strengths
- The partial/full distinction is motivated and supported by relevant results.
The submission goes beyond treating every occluded response as missing information. The measured difference between partial- and full-occlusion discrimination provide a concrete reason to retain the former. The resulting likelihood is simple enough that its behavior at the limiting cases can be understood directly.
- The comparisons isolate the fusion question reasonably well.
Tracks and cached semantic responses are held fixed, and the experiments include ordinary censoring, validity-weighted likelihood ratios, and hard censoring. The additional two-component comparison on Confirmation 2 is particularly useful, which shows that ordinray censoring alone does not explain the reported improvement.
- The benchmark supports a useful diagnostic task.
Persistent surface identity, geometric visibility checks, and partial/full labels allow questions that object-level grounding annotations cannot directly answer. The authors also take care not to present target-track selection as full object localization, or recovered-support ranking as complete surface coverage.
Critical weaknesses
- The strongest evidence remains confined to synthetic scenes with analytic support.
The primary gains assume that the relevant surfaces and their projections are already available. In the fully estimated setting, target-track recall is only 0.39, and the coverage-aware evaluation report differences of roughly 0.003 between fusion rules. On ScanNet++, windowed averaging is better than every tested likelihood rule, which do not invalidate the controlled finding, but they leave the practical grounding contribution substantially less established than the diagnostic contribution.
- The comparison with learned fusion weakens the case for the proposed model beyond a low-data setting.
At full supervision, the additive MLP is ahead in 5/10 SigLIP runs and 7/10 SD runs, with most remaining runs unresolved. Therefore, the structured model’s demonstrated advantage is mainly in the very-low-data regime, where it also fails to fit on two of five one-scene subsets. This is acknowledged, but the main experimental presentation should give these results more prominence, and reports absolute performance alongside the likelihood baselines.
- The spatial gate is useful empirically but insufficiently characterized.
Taking the maximum estimated visibility within a radius tied to the entire track-set extent can admit evidence from neighboring objects, and may be sensitive to scene scale, track density, and outliers. Non-inferiority to the instance-grouped gate on the synthetic confirmation set do not establish robustness to these factors. A radius sweep and comparisons with simpler gates would help to establish whether the spatial construction itself is necessary.
- The partial/full ablation does not fully isolate the source of the gain.
The refinement changes the response densities, which introduces class-conditional censored evidence, and add a spatial gate. Its improvement over the two-component model establishes the benefit of the complete refinement, but not whether accurate partial/full separation is essential. Constant-gate and shuffled-gate controls would help to distinguish meaningful spatial conditioning from the benefit of a more flexible response model.
Questions for authors
- What exactly supplies the validity cues in the analytic-track experiments?
Please specify which quantities come from analytic geometry, and which come from estimated geometry, including the construction of depth margins and point-map residuals. I suggest that a dependency table would make the test-time assumptions easier to assess.
- How much does the spatial gate contribute beyond a globally fitted mixture?
Could the authors compare the current gate with a development-selected constant , , and a shuffled spatial gate, while keep the response densities fixed? How sensitive are the results to the neighborhood radius?
- Is the shared full-occlusion density preferable to a regularized class-conditional alternative?
Full-occlusion AUROC of 0.45 under SigLIP suggests some inverse association, even if weak. A comparison with separately fitted full-occlusion densities would clarify whether sharing is beneficial regularization, or it may discard useful information.
4 comments
Thank the author for the careful responses and more experiments. I have carefully read your rebuttal. Most of my key concerns are addressed, like the geometric cue dependencies, the rationale for sharing the full-occlusion density, and the contribution of the spatial gate beyond a constant mixture.
While some concerns remain regarding the limited evidence under estimated geometry, and the scope of the low-data advantage. In particular, the reported low-data results use the instance-grouped gate and are conditional on successful fitting. Is it possible to provide a corresponding low-data comparison using the grouping-free spatial gate, reporting both performance and fitting success rates? which help to clarify whether the data-efficiency benefit extends to the proposed inference setting.
Despite these remaining limitations, I think that the occlusion analysis and the diagnostic benchmark are valuable contributions, and the rebuttal has addressed most of my concerns. Thus, I raise my rating from 2 to 3.
Additional Low-Data Results with the Grouping-Free Spatial Gate and Fitting Success Rates
Reply to the Area Chair and Reviewer DAJw: spatial-gate low-data comparison and fitting success
Thank you for identifying the remaining question. Summary: with the grouping-free gate, the one-scene mixture advantage is +0.032 (SD) and +0.012 (SigLIP), with 46/48 mixture fits; SigLIP results vary substantially across development scenes. We ran a post-hoc low-data comparison with the submitted grouping-free spatial gate (γ=0.06) on the Appendix D population (48 development scenes; 120 test scenes/672 episodes; analytic tracks). Both models use the spatial gate in evaluation, and the MLP uses it during training. The original oracle-grouped results were reproduced on their original subsets before this extension.
We evaluated all 48 possible one-scene development subsets, 30 sampled subsets each at 2 and 5 scenes, and the original five subsets each at 10 and 25 scenes. The MLP uses 3 training seeds per subset at 1–5 scenes and 5 seeds at 10–25 scenes; the 48-scene row uses the previously reported 10-seed comparison. Entries are mean AUROC_occ over subsets where both models can be evaluated, with MLP scores averaged over seeds. Differences use unrounded scores. The final column counts subsets whose mixture-minus-MLP paired 95% test-scene-bootstrap intervals are positive / overlap zero / negative. These exploratory counts are not multiplicity-adjusted.
Stable Diffusion (SD)
| Dev scenes | Mixture fitted | Mixture / MLP | Difference | Mixture ahead / unresolved / MLP ahead |
|---|---|---|---|---|
| 1 | 46/48 | 0.856 / 0.825 | +0.032 | 40 / 6 / 0 |
| 2 | 30/30 | 0.865 / 0.837 | +0.028 | 27 / 3 / 0 |
| 5 | 30/30 | 0.868 / 0.853 | +0.015 | 26 / 4 / 0 |
| 10 | 5/5 | 0.871 / 0.863 | +0.008 | 3 / 2 / 0 |
| 25 | 5/5 | 0.872 / 0.868 | +0.004 | 2 / 3 / 0 |
| 48 | 1/1 | 0.872 / 0.877±0.003 | −0.005 | — |
SigLIP
| Dev scenes | Mixture fitted | Mixture / MLP | Difference | Mixture ahead / unresolved / MLP ahead |
|---|---|---|---|---|
| 1 | 46/48 | 0.899 / 0.888 | +0.012 | 27 / 10 / 9 |
| 2 | 30/30 | 0.911 / 0.901 | +0.010 | 16 / 11 / 3 |
| 5 | 30/30 | 0.919 / 0.921 | −0.002 | 5 / 14 / 11 |
| 10 | 5/5 | 0.924 / 0.925 | −0.001 | 0 / 4 / 1 |
| 25 | 5/5 | 0.924 / 0.927 | −0.003 | 0 / 1 / 4 |
| 48 | 1/1 | 0.926 / 0.929±0.001 | −0.003 | — |
Fitting success and its denominator. The mixture fits on 46/48 (95.8%) single-scene choices; the MLP trains on 48/48. The two mixture failures are the two scenes containing no fully occluded observations, leaving the shared full-occlusion density h without fitting data. Both failures happened to occur in the paper's original five one-scene draws, yielding its unrepresentative 3/5 fit count. For those two scenes, the MLP scores 0.809/0.812 (SD) and 0.865/0.870 (SigLIP); these are not included in the paired means. All evaluated subsets at 2, 5, 10 and 25 scenes fit successfully. A separate fit-only screen of 200 sampled subsets at each of 2, 5 and 10 scenes also found 200/200 valid mixture fits per size; these extra subsets are not included in the performance tables. The reported one-scene performance means are conditional on the 46 jointly evaluable subsets, not on all 48.
Interpretation. The low-data advantage survives removal of oracle instance grouping, but is backbone-dependent. Under SD, the mixture leads on 40/46 one-scene and 26/30 five-scene subsets, and is never significantly behind on any evaluated subset with 1–25 scenes. Its mean advantage decreases with more supervision; the MLP is slightly ahead at 48 scenes. Under SigLIP, the mean one-scene margin is only +0.012, with the mixture ahead on 27/46 subsets and the MLP ahead on 9/46; the MLP's mean overtakes by five scenes. On the paper's original three fittable one-scene subsets, SigLIP's margin was +0.032 with the grouped gate (MLP seed 0) and +0.031 with the spatial gate (also +0.031 using seed 0 alone or averaging seeds 0–2, as in the table). In the three-seed spatial analysis, these subsets ranked 6th, 9th and 10th among 46 by margin. Their larger gains than the 46-subset mean reflect which random scenes were drawn, not evidence of a substantial grouped-gate advantage.
We will foreground these spatial-gate comparisons and fitting denominators, retain the oracle-grouped results as diagnostics, replace the paper's "ahead in every repeat" claim with the subset counts, qualify the single-scene fit requirement, and acknowledge that the MLP is competitive or slightly better at full supervision. These are exploratory analyses on synthetic analytic support and a previously examined test split; per-subset bootstrap intervals quantify test-scene sampling uncertainty, not uncertainty over development-subset choice or training seeds.
Dear authors, thank you for your detailed responses. The low-data comparisons using the grouping-free spatial gate, together with the fitting success rates, answer my remaining question. The reporting of the backbone-dependent gains and fitting limitations is good. I hope these qualifications and the complete results can be clearly reported into the revised manuscript.
Gate controls, learned fusion comparisons, and geometric cue dependencies
Thank you for the suggestions. New Confirmation 2 analyses use the same 160 scenes/882 episodes per backbone and cached inputs. They are post hoc; intervals are paired scene-bootstrap 95%. All AUROCs are AUROC_occ; paired entries are SD / SigLIP. Differences use unrounded metrics.
W1. Scope. The strongest evidence concerns controlled surface ranking and target-track selection. Estimated geometry has target-track recall 0.39 and coverage-aware fusion differences near 0.003. We will foreground these limits and the readout dependence of real-data results.
W2. Learned fusion. App. D uses the 120-scene/672-episode core population and oracle-grouped ρ. The new spatial comparison uses the submitted gate during both training and evaluation on the same population.
Subset rows are means; full-data MLP entries are mean±SD over 10 seeds. The best non-censored core rules reach 0.824 / 0.883. The MLP has higher full-data means. The low-data evidence uses grouped ρ and is conditional on fitting success: two of five one-scene subsets fail. The new full-data run does not establish a spatial low-data advantage. We will foreground these absolute results.
W3–W4/Q2. Gate controls and radius. Three-component controls share f0,f1,g0,g1,h and all other inputs; only ρ changes. The two-component reference retains its pooled censored density c.
The constant gate recovers most of the AUROC gain over two-component censoring, but its R@1 point estimates are lower. Spatial-minus-constant AUROC is about +0.010 [+0.007, +0.014] for each backbone; R@1 gains are +2.5 pp [+0.7, +4.4] / +6.9 pp [+4.7, +9.2].
The shuffle uniformly permutes whole per-track gate vectors within each scene, without stratification (five permutations; metrics averaged), holding η, frustum masks and densities fixed. It preserves each view's raw gate distribution over all tracks, but changes its alignment with validity and scored observations; ρ≥η is not reimposed. The 0.022 / 0.024 AUROC loss therefore reflects this combined disruption, rather than isolating spatial conditioning.
AUROC varies by about 0.003 across γ=0.06–0.10 and decreases at smaller or much larger radii. At γ=0.06, neighborhoods average 39 tracks, 6% from other instances. These measurements do not establish robustness to changed track density, outliers or scene composition.
Q1. Cue dependencies. Ground truth supplies analytic tracks x_t, cameras and projections. VGGT supplies depth D̂_i, depth confidence, point maps P̂_i and predicted camera extrinsics. Let S be the similarity fitted from VGGT camera centers to ground-truth centers. We use its inverse, X̃_t=S⁻¹(x_t), to express each analytic track in VGGT's frame; point maps are unchanged.
Let p_ti be the nearest-pixel ground-truth projection, and z_ti the depth of X̃_t under VGGT's predicted extrinsics for view i. P̂_i(p_ti) and X̃_t use the same reconstruction frame. For positive z_ti, the raw cues are m_ti=(D̂_i(p_ti)−z_ti)/z_ti and r_ti=‖P̂_i(p_ti)−X̃_t‖₂/z_ti. A negative margin means that the sampled predicted depth is smaller than the track's estimated camera depth.
A logistic predictor uses these cues and depth confidence, with clipping and development-set median/IQR standardization; a histogram calibrator yields η. Both models are fitted on labeled development scenes. Primary spatial inference uses no test validity, membership, type or instance-group labels; oracle-grouped diagnostics use instance groups. Estimated validity still depends on analytic geometry. Fully estimated inference uses VGGT tracks and cues in its own frame; ground truth supplies development supervision and evaluation labels, not test-time inputs.
Q3. Shared full-occlusion density. With bins and other components fixed, we fitted separate h0,h1 and interpolated toward shared h. Shared-minus-separate AUROC gains are +0.0025 [+0.0015,+0.0037] / +0.023 [+0.014,+0.037]; under SigLIP, full-occlusion negative evidence rises from 0.016 to 0.199 nats with separate fits. The shared endpoint performs best among the tested settings, supporting regularization within this density family rather than absence of usable full-occlusion information.