- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how CLIP features lifted into 3D Gaussian scenes are compared with text at query time. The method adjusts visual and text vectors using the mean direction of scene features and fits a text-side transform using a fixed 45-word reference bank; multiclass prediction uses soft clustering, and single-query prediction uses a CSLS-corrected margin against the reference words. The authors report mIoU gains relative to raw feature comparison across four indoor datasets and seven feature lifting methods, without retraining or re-lifting features.
Critical strengths
-
The paired experiments on fixed feature fields have practical value. Gains are observed across multiple lifting methods and datasets, suggesting that improving how existing feature fields are used is a worthwhile research direction.
-
Multiclass prediction and single-query rejection are treated separately. Rather than substituting single-query evaluation with multiclass
argmaxperformance, the authors design a reference-word competition margin for the case where only a single target word is provided, and report results separately.
Critical weaknesses
-
The experiments cannot yet fully attribute the gains to the geometric calibration mechanism emphasized by the paper. The directional concentration and ground-truth-labeled class-separation diagnostics show that the transformation changes feature geometry, but they cannot establish that the segmentation gains of the full method mainly come from scene centering or text whitening. In particular, in the ScanNet++ multiclass SFS ablation, soft clustering alone already raises mIoU from 19.92 to 23.66, the full method reaches 23.74, whereas "Feature Calibration" alone—which combines centering and whitening—drops to 18.77 (Table 3). The current ablations do not separate centering, reference-word mean adjustment, and spectral scaling transform. Since the shared scene direction may itself carry class information, the current evidence is insufficient to support the stronger mechanistic interpretation that this direction should be universally subtracted.
-
The background rejection capability, which is most critical for the single-query branch, has not yet been validated. The experiments only query, one by one, the classes that actually exist in a scene, and use thresholds selected on held-out scenes; the authors also explicitly acknowledge that this cannot prove reliable rejection of entirely absent targets or arbitrary natural-language queries (§4.2, §5). The bank only avoids exact name matches; broad semantic overlap is explicitly not excluded (Appendix B). Moreover, Figure 4 in Appendix A, used to display the response maps, uses , whereas the main experiments use , and the Raw and calibrated maps are normalized independently; these figures cannot make up for the lack of evidence on threshold-based rejection under the main setting (Appendix A, D). This gap directly limits the scope of the paper's single-query contribution.
Questions for authors
-
Please add one component-wise ablation table under the same SFS fields, scenes, and calibration protocol as Table 3, with rows for raw, centering only, reference-mean adjustment only, spectral scaling only, soft clustering only, and the full method, reporting ScanNet and ScanNet++ mIoU. Please also compare the selected 45-word bank against the original 64-word pool and an equal-size unfiltered bank. These results will determine whether I accept the paper's explanation of the source of the gains and of the role of the fixed vocabulary.
-
Could classes absent from each scene be selected from the existing evaluation vocabulary, keeping the thresholds of Appendix C unchanged, and report the false-positive primitive/pixel ratio for each absent query, while also reporting IoU for the present classes? In particular, please list absent queries that are semantically close to reference words or to classes already present in the scene. This requires no feature re-lifting, yet directly tests the background-rejection use case claimed by the paper.
-
Why does Figure 4 use parameters different from those of the main experiments? Could examples of binary predictions be provided under the main experimental parameters using the actual thresholds? These clarifications will affect the interpretation of the qualitative evidence and of the cross-method results.
6 comments
Parameter inconsistencies and thresholded masks [W5, Q3]
We apologize for the confusion around lambda and alpha parameters in Figure 4. We forgot to update the text in the appendix after changing the parameter selection from multi-class performance on held-out scenes to a mix of multi and single-class performance, the procedure described in Appendix D. We confirm that Figure 4 is produced with and correct the manuscript accordingly.
Similarly, following our calibration protocol (see Common Response CR1), our thresholds are fixed and we have added according binary mask results to our Appendix, please see Figure 5 in the revised manuscript.
I have reviewed your detailed responses, which resolve my earlier concerns, leaving one final question on single-query rejection.
While the Presence Gate substantially reduces absent-query foreground responses, average Gaussian-level FP% can obscure how often an absent query still produces a mask.
Since the gate starts from raw cosine scores, could you apply the same gate (top-1% candidate selection, 0.5 cutoff) to the Raw baseline across all four datasets while keeping each method’s existing segmentation threshold unchanged? For both Raw + Gate and Ours + Gate, please report present-query mIoU (counting rejected present queries as empty masks), Gaussian-level FP%, and the query-level false acceptance rate—defined as the fraction of absent queries yielding a non-empty final mask under the same criterion (briefly stated). A clear mIoU gain for Ours + Gate at a comparable or lower query-level false acceptance rate would resolve my remaining concern; if the rates differ substantially, comparing near a shared false acceptance rate using held-out thresholds would verify whether the localization gain holds under similar rejection.
We thank the reviewer for their constructive feedback. We have submitted a revised manuscript with changes marked in blue. The Common Response addresses the main concerns shared across reviewers. Below, we respond to this reviewer’s specific weaknesses and questions.
Feature calibration vs. score calibration and fine grained ablations [W1, W2, Q1a]
We agree with the reviewer that geometric calibration alone cannot be attributed the full increase in segmentation gains in our work and we argue in the paper that both the feature calibration and the score calibration are integral components that complement each other. To investigate the individual contributions closer we extend our ablation study to the fine grained components within our feature calibration and extend the ablation to all four datasets. Please find the experiment in our Common Response CR3.
Mechanistic interpretation. We do not assume that the shared scene direction is nonsemantic. It may encode scene context shared by many Gaussians, which can be redundant for distinguishing objects within that scene. Attenuating this common component is therefore motivated by an intuition about within-scene discrimination, rather than by a claim that the direction contains no useful information. We have not directly established this causal interpretation. Our evidence for our presented inuition is the segmentation effect measured by the component ablations. We do not claim that removing a shared direction must be beneficial in every scene or setting.
Absent queries [W3, W4, Q2]
We follow the requested evaluation by selecting classes from the original vocabulary that are absent from each scene’s annotations and applying the unchanged main-evaluation thresholds for Raw and calibrated predictions. The Common Response CR2 reports dataset-level absent-query foreground ratios together with present-query mIoU. Here we examine individual-query behavior.
Distribution across absent queries. The following table summarizes all 626 D1 scene–query pairs. FP% is the percentage of evaluated Gaussians predicted as foreground. Quantiles weight each scene–query pair equally.
Calibration increases the median absent-query response on both datasets, although the maximum response decreases. Thus, improved present-query localization does not imply reliable rejection of absent targets. Appendix Table 16 additionally reports the five largest calibrated responses, with Raw values alongside them.
Absent queries close to reference words. To address the requested semantic-overlap examples, the following table combines individual foreground ratios with each query’s nearest reference word in the original CLIP text space. These are illustrative annotation-absent cases: table and cloth show intuitive reference relationships, book has higher text similarity but a small foreground response, sink shows the largest Raw-to-Ours D1 FP increase, at lower text similarity.
scene0400_0009c1414f1bf9f95681fdscene0400_00Reference bank selection [Q1b]
We compare the selected G45 bank with the original G64 pool and three random subsets of equal size. G45 was selected using the held-out ScanNet and ScanNet++ scenes and then fixed across evaluation datasets. It outperforms G64 on all four datasets and all three sampled random banks on three datasets, but on LERF-OVS, one random bank performs better. Thus, the selected bank transfers without per-dataset reselection but is not universally optimal. The strong results obtained with alternative banks also indicate that the calibration gains do not depend exclusively on the particular selected vocabulary. Given these limitations of our selection strategy, exploring better vocabulary selection mechanisms is a promising direction for future work.