acceptodds
Under review as a conference paper at ICLR 2027

Query-Time Calibration of CLIP Features for Open-Vocabulary 3D Segmentation

Abstract

Lifting vision-language features such as CLIP into 3D Gaussian reconstructions enables open-vocabulary segmentation through natural-language queries. Although CLIP represents a broad range of semantic concepts, the lifted features of individual scenes can exhibit strong directional concentration, while raw cosine similarity does not account for the scene-specific feature distribution. We show that substantial segmentation gains can be obtained by changing how a fixed lifted feature field is compared with text. Our method partially subtracts a shared scene direction from visual features and text prototypes and whitens text prototypes using a fixed reference vocabulary of generic terms. It then uses soft clustering for multiclass prediction and CSLS with a relevance margin against the reference vocabulary for single-query prediction. Beyond reducing directional concentration, scene centering improves class-relative feature separation in every evaluated scene. Across seven lifting methods on ScanNet and ScanNet++, the resulting query-time calibration consistently improves mean intersection over union in both multiclass and single-query segmentation. These gains require no retraining, source-image re-encoding, or feature re-lifting, demonstrating the value of scene-conditioned comparison for making better use of existing lifted representations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary segmentation, gaussian splatting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dvxBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how CLIP features lifted into 3D Gaussian scenes are compared with text at query time. The method adjusts visual and text vectors using the mean direction of scene features and fits a text-side transform using a fixed 45-word reference bank; multiclass prediction uses soft clustering, and single-query prediction uses a CSLS-corrected margin against the reference words. The authors report mIoU gains relative to raw feature comparison across four indoor datasets and seven feature lifting methods, without retraining or re-lifting features.

Critical strengths
  1. The paired experiments on fixed feature fields have practical value. Gains are observed across multiple lifting methods and datasets, suggesting that improving how existing feature fields are used is a worthwhile research direction.

  2. Multiclass prediction and single-query rejection are treated separately. Rather than substituting single-query evaluation with multiclass argmax performance, the authors design a reference-word competition margin for the case where only a single target word is provided, and report results separately.

Critical weaknesses
  1. The experiments cannot yet fully attribute the gains to the geometric calibration mechanism emphasized by the paper. The directional concentration and ground-truth-labeled class-separation diagnostics show that the transformation changes feature geometry, but they cannot establish that the segmentation gains of the full method mainly come from scene centering or text whitening. In particular, in the ScanNet++ multiclass SFS ablation, soft clustering alone already raises mIoU from 19.92 to 23.66, the full method reaches 23.74, whereas "Feature Calibration" alone—which combines centering and whitening—drops to 18.77 (Table 3). The current ablations do not separate centering, reference-word mean adjustment, and spectral scaling transform. Since the shared scene direction may itself carry class information, the current evidence is insufficient to support the stronger mechanistic interpretation that this direction should be universally subtracted.

  2. The background rejection capability, which is most critical for the single-query branch, has not yet been validated. The experiments only query, one by one, the classes that actually exist in a scene, and use thresholds selected on held-out scenes; the authors also explicitly acknowledge that this cannot prove reliable rejection of entirely absent targets or arbitrary natural-language queries (§4.2, §5). The bank only avoids exact name matches; broad semantic overlap is explicitly not excluded (Appendix B). Moreover, Figure 4 in Appendix A, used to display the response maps, uses (λ,α)=(0.45,0.60)(\lambda,\alpha)=(0.45,0.60), whereas the main experiments use (0.30,0.40)(0.30,0.40), and the Raw and calibrated maps are normalized independently; these figures cannot make up for the lack of evidence on threshold-based rejection under the main setting (Appendix A, D). This gap directly limits the scope of the paper's single-query contribution.

Questions for authors
  1. Please add one component-wise ablation table under the same SFS fields, scenes, and calibration protocol as Table 3, with rows for raw, centering only, reference-mean adjustment only, spectral scaling only, soft clustering only, and the full method, reporting ScanNet and ScanNet++ mIoU. Please also compare the selected 45-word bank against the original 64-word pool and an equal-size unfiltered bank. These results will determine whether I accept the paper's explanation of the source of the gains and of the role of the fixed vocabulary.

  2. Could classes absent from each scene be selected from the existing evaluation vocabulary, keeping the thresholds of Appendix C unchanged, and report the false-positive primitive/pixel ratio for each absent query, while also reporting IoU for the present classes? In particular, please list absent queries that are semantically close to reference words or to classes already present in the scene. This requires no feature re-lifting, yet directly tests the background-rejection use case claimed by the paper.

  3. Why does Figure 4 use parameters different from those of the main experiments? Could examples of binary predictions be provided under the main experimental parameters using the actual thresholds? These clarifications will affect the interpretation of the qualitative evidence and of the cross-method results.

6 comments

We thank the reviewer for their constructive feedback. We have submitted a revised manuscript with changes marked in blue. The Common Response addresses the main concerns shared across reviewers. Below, we respond to this reviewer’s specific weaknesses and questions.

Feature calibration vs. score calibration and fine grained ablations [W1, W2, Q1a]

We agree with the reviewer that geometric calibration alone cannot be attributed the full increase in segmentation gains in our work and we argue in the paper that both the feature calibration and the score calibration are integral components that complement each other. To investigate the individual contributions closer we extend our ablation study to the fine grained components within our feature calibration and extend the ablation to all four datasets. Please find the experiment in our Common Response CR3.

Mechanistic interpretation. We do not assume that the shared scene direction is nonsemantic. It may encode scene context shared by many Gaussians, which can be redundant for distinguishing objects within that scene. Attenuating this common component is therefore motivated by an intuition about within-scene discrimination, rather than by a claim that the direction contains no useful information. We have not directly established this causal interpretation. Our evidence for our presented inuition is the segmentation effect measured by the component ablations. We do not claim that removing a shared direction must be beneficial in every scene or setting.

Absent queries [W3, W4, Q2]

We follow the requested evaluation by selecting classes from the original vocabulary that are absent from each scene’s annotations and applying the unchanged main-evaluation thresholds for Raw and calibrated predictions. The Common Response CR2 reports dataset-level absent-query foreground ratios together with present-query mIoU. Here we examine individual-query behavior.

Distribution across absent queries. The following table summarizes all 626 D1 scene–query pairs. FP% is the percentage of evaluated Gaussians predicted as foreground. Quantiles weight each scene–query pair equally.

DatasetMethodPairsMedian FP%90th-percentile FP%Maximum FP%
ScanNetRaw931.1320.4264.59
ScanNetOurs9311.5620.2936.46
ScanNet++Raw5330.112.8329.47
ScanNet++Ours5332.196.3418.79

Calibration increases the median absent-query response on both datasets, although the maximum response decreases. Thus, improved present-query localization does not imply reliable rejection of absent targets. Appendix Table 16 additionally reports the five largest calibrated responses, with Raw values alongside them.

Absent queries close to reference words. To address the requested semantic-overlap examples, the following table combines individual foreground ratios with each query’s nearest reference word in the original CLIP text space. These are illustrative annotation-absent cases: table and cloth show intuitive reference relationships, book has higher text similarity but a small foreground response, sink shows the largest Raw-to-Ours D1 FP increase, at lower text similarity.

Dataset / sceneAbsent queryNearest referenceText cosineRaw FP%Ours FP%
ScanNet / scene0400_00tablefurniture0.8035.1614.56
ScanNet++ / 09c1414f1bclothpattern0.8582.181.96
ScanNet++ / f9f95681fdbookprofile0.8832.102.20
ScanNet / scene0400_00sinkvolume0.7540.4923.01

Reference bank selection [Q1b]

We compare the selected G45 bank with the original G64 pool and three random subsets of equal size. G45 was selected using the held-out ScanNet and ScanNet++ scenes and then fixed across evaluation datasets. It outperforms G64 on all four datasets and all three sampled random banks on three datasets, but on LERF-OVS, one random bank performs better. Thus, the selected bank transfers without per-dataset reselection but is not universally optimal. The strong results obtained with alternative banks also indicate that the calibration gains do not depend exclusively on the particular selected vocabulary. Given these limitations of our selection strategy, exploring better vocabulary selection mechanisms is a promising direction for future work.

Reference bank / MC mIoU (%)ScanNetScanNet++LERF-OVS3D-OVS
Selected G4539.3723.7465.6789.80
Original G6437.7722.9364.8089.44
Random45 — seed 138.0023.1165.3786.83
Random45 — seed 236.2722.4565.1287.72
Random45 — seed 337.3221.7366.2688.78
Random45 mean ± SD37.19 ± 0.7122.43 ± 0.5665.58 ± 0.4887.7 ± 0.79

Parameter inconsistencies and thresholded masks [W5, Q3]

We apologize for the confusion around lambda and alpha parameters in Figure 4. We forgot to update the text in the appendix after changing the parameter selection from multi-class performance on held-out scenes to a mix of multi and single-class performance, the procedure described in Appendix D. We confirm that Figure 4 is produced with (λ,α)=(0.3,0.4)(\lambda, \alpha) = (0.3, 0.4) and correct the manuscript accordingly.

Similarly, following our calibration protocol (see Common Response CR1), our thresholds are fixed and we have added according binary mask results to our Appendix, please see Figure 5 in the revised manuscript.

I have reviewed your detailed responses, which resolve my earlier concerns, leaving one final question on single-query rejection. While the Presence Gate substantially reduces absent-query foreground responses, average Gaussian-level FP% can obscure how often an absent query still produces a mask. Since the gate starts from raw cosine scores, could you apply the same gate (top-1% candidate selection, 0.5 cutoff) to the Raw baseline across all four datasets while keeping each method’s existing segmentation threshold unchanged? For both Raw + Gate and Ours + Gate, please report present-query mIoU (counting rejected present queries as empty masks), Gaussian-level FP%, and the query-level false acceptance rate—defined as the fraction of absent queries yielding a non-empty final mask under the same criterion (briefly stated). A clear mIoU gain for Ours + Gate at a comparable or lower query-level false acceptance rate would resolve my remaining concern; if the rates differ substantially, comparing near a shared false acceptance rate using held-out thresholds would verify whether the localization gain holds under similar rejection.

Reviewer_4mbROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free, query-time calibration for CLIP features lifted onto 3D Gaussians. It partially subtracts the scene's mean feature direction from visual features and text prototypes, whitens text prototypes using a fixed reference vocabulary of generic terms, then applies TransCLIP-style soft clustering for multi-class prediction and a CSLS-corrected margin over the reference bank for single-query prediction. Across seven lifting methods and four datasets (LERF-OVS, 3D-OVS, ScanNet, ScanNet++), mIoU improves over raw cosine scoring, by up to +19 on 3D-OVS and +12.5 on ScanNet for SFS, but by only +1 to +4 on ScanNet++.

Critical strengths

This paper proposes a cheap and validated fix. The calibration needs no retraining, re-encoding or re-lifting, and is evaluated on seven lifting methods, four datasets and two query settings with precision/recall reported, held-out hyperparameter selection, component ablations, and a comparison against four adapted calibration baselines.

Critical weaknesses
  1. Most of the multi-class gain come from TransCLIP's soft clustering, not the new components. Table 3 shows score calibration alone gives +8.2 of +12.5 on ScanNet and +3.7 of +3.8 on ScanNet++, and full TransCLIP in Table 2 already gives +8.7/+2.9. Centering and bank whitening therefore add about +4 and +1. On LERF-OVS and 3D-OVS the multi-class vocabulary is each scene's present classes, and single-query evaluation only queries present targets, so rejection of absent classes is largely untested.

  2. The raw baseline is not the methods' native query pipeline, and two of seven baselines are degenerate. Each method has its own scoring, and the paper replaces it with raw cosine and never compares with a published number. On 3D-OVS, Occam's own pipeline reports 95.0 vs. 84.2 calibrated here, so calibration may repair weak scoring rather than add to strong pipelines, which is never tested.

Questions for authors
  1. Which scenes are the held-out calibration set, and are they disjoint from all evaluation scenes?
  2. What does full TransCLIP plus your centering and bank give on all datasets?
  3. Does calibration add to a method's native query pipeline, e.g., Occam's full pipeline or LangSplat with level selection?
  4. For multi-class, what happens when absent distractor classes are added to the vocabulary?
4 comments

We thank the reviewer for their constructive feedback. We have submitted a revised manuscript with changes marked in blue. The Common Response addresses the main concerns shared across reviewers. Below, we respond to this reviewer’s specific weaknesses and questions.

Contributions beyond soft clustering [W1]

We agree with the reviewer that the soft clustering has a strong contribution to our overall gains, however it does not always account for the majority of the gains and only applies in the multi-class case. We find on some datasets score calibration provides the majority gain, while on others feature calibration has the bigger impact. Most notably, we find that they are complimentary and their combination consistently leads to the best result. To highlight this we extend our ablation study to all four datasets in our Common Response CR3

Rejection of absent queries [W2]

We thank the reviewer for pointing this out. Absent queries are an interesting aspect that we have not considered in our work. We respond to this in our Common Response CR2.

Raw baseline vs. native query pipeline (1/2) [W3]

We agree that Raw is not a reproduction of each lifting method’s complete native inference pipeline. Our main comparison holds the lifted feature field fixed and uses a common cosine readout to isolate the effects of calibration. Different lifting methods are included to assess compatibility with their learned fields, rather than to rank their complete native systems. We will make this distinction explicit in the manuscript.

In particular, our fixed-field Occam evaluation differs from its native pipeline in reconstruction, field-level selection, and scoring. We therefore distinguish two additional comparisons: an isolated comparison with Occam’s four-negative scoring on the same SFS fields, and a separate integration into Occam’s multilevel pipeline on 3D-OVS. The former tests alternative scoring under controlled fields, while the latter tests compatibility with a more complete system.

Occams Scoring mechanism. We compare Occams score calibration as an isolated component to our calibration, matching our Table 3 (previously Table 2) setup using the SFS field. Occam proposes comparing the query to negative queries "object", "things", "stuff", "texture", applies a temperature-softmax to each query-negative pair and retains the lowest query probability as relevance (four-negative relevance). In the case of LERF and 3D-OVS where IoU is evaluated in image space, cosine scores are rendered and relevance is computed in image space. Further Occam proposes to minmax normalize relevance across the resulting images, which we adapt to normalize over all Gaussians for ScanNet and ScanNet++ which are evaluated in 3D. The effect of the four-negative scoring depends on the normalization and evaluation domain. Its normalized variant is particularly effective on the image-evaluated datasets. Our calibration performs better on the two 3D-evaluated datasets, while the normalized four-negative readout is slightly stronger on LERF-OVS and 3D-OVS.

Scoring / N=1ScanNetScanNet++LERF-OVS3D-OVS
Raw cosine26.6622.5545.7169.21
Four-negative relevance24.8319.2745.9570.47
Four-negative relevance + minmax26.8721.8554.3784.59
Full calibration34.5224.3953.8683.59

Raw baseline vs. native query pipeline (2/2) [W3]

Adding calibration to Occam's full system. [W5, Q3] We found investigating our calibrations in combination with level selection out of scope for our paper and focused on a single fixed feature field, because level selection would open another axis of possible options, while not all existing selection mechanisms transfer cleanly to our calibration. Nevertheless, we present one possible adaption here for Occam on the 3D-OVS dataset, since the reviewer had specific questions about this number. Please note that reconstructing all additional levels for all our datasets takes too long in the scope of this rebuttal.

Below we present our reproduction of Occam's full pipeline on the 3D-OVS dataset with mIoU on the individual levels, as well as final mIoU using Occams level selection. Note that we could not reproduce their claimed 95.0 exactly. We used their officially provided code and report the resulting numbers. For Occam + FC we modified their pipeline by applying our scene centering and text whitening to their features independently at each level, retaining their scorer. In Occam + SC we use the raw features and apply our score calibration, i.e. soft clustering on each level. For level selection we use for each query the cluster with the lowest feature-space cosine dispersion. Lastly Occam + FC + SC does the same with calibrated features. The results show that feature calibration alone does not substantially change Occam's performance, while score calibration leads to a slight increase. The combined adaptation gives the largest merged improvement with a 2.752.75 IoU gain and fixed level results provide complementary evidence, where FC+SC significantly raises L1 and L2 scores. These results demonstrate that our calibration can be a useful addition to a full scoring system like Occam's.

MethodMergedL1L2L3
Occam87.1361.0781.4586.56
Occam + FC86.9065.9681.7485.71
Occam + SC88.1269.4182.1386.77
Occam + FC + SC89.8878.8786.7886.19

Weak lifted fields [W4]

We agree that LangSplat’s fixed-field results and the LUDVIG reproduction on 3D-OVS are very weak. Our LangSplat adaptation omits native level selection and spatial smoothing, while the LUDVIG result is explicitly flagged as a reproduction limitation. We therefore do not rely on these cases to establish the effectiveness of calibration. The evidence also covers five other lifting methods across four datasets, together with LUDVIG on the remaining datasets. These limitations reduce the breadth of informative fields in particular cases, rather than invalidate the matched Raw-versus-calibrated comparisons on the remaining fields.

Held-out calibration set [Q1]

We clarify our calibration protocol in the Common Response CR1.

Our evaluation scenes are disjoint from the calibration scenes.

Calibration scenes: scene0697_02 (ScanNet) and fd361ab85f (ScanNet++)

Evaluation Scenes:

ScanNet: scene0000_00, scene0070_00, scene0140_00, scene0347_00, scene0590_00, scene0062_00, scene0097_00, scene0200_00, scene0400_00, scene0645_00

ScanNet++: 09c1414f1b, 0d2ee665be, 27dd4da69e, 3864514494, 3db0a1c8f3, 578511c8a9, 5942004064, 9071e139d9, c50d2d1d42, d755b3d9d8, e7af285f7d, f9f95681fd

Fine grained TransCLIP comparison [W1, Q2]

Using TransCLIP's graph does not reliably improve accuracy, while graph construction averages roughly 500 s on our two-scene 3D-OVS sample and cached inference takes up to 2.3× the graph-free time.

Scoring / MC mIoU (%)Feature calibrationScanNetScanNet++LERF-OVS3D-OVS
Graph-free soft clusteringOff35.0923.6654.3585.88
Graph-free soft clusteringOn39.3723.7465.6789.80
Full-graph TransCLIPOff35.5423.8056.7185.87
Full-graph TransCLIPOn39.7523.6365.3589.56

Distractors in multi-class [W2, Q4]

Adding distractors reveals dataset-dependent sensitivity to the candidate vocabulary. Calibration retains an mIoU advantage on ScanNet, ScanNet++, and LERF-OVS, whereas the original advantage on 3D-OVS is substantially reduced and is not preserved when both distractor groups are added. Thus, our improved multi-class segmentation results using the original benchmark vocabularies does not imply reliable rejection of arbitrary absent candidates. We investigate query-level absence separately in the single-query setting, where our proposed presence gate provides a practical localization–rejection trade-off.

ScanNet vocabularyRaw mIoU (%)Calibrated mIoU (%)Raw absent assignment (%)Calibrated absent assignment (%)
Original (incl. D1)26.8739.3713.5022.91
+D226.2136.8316.7834.16
+D326.8639.7113.8926.11
+D2+D326.2136.8017.1635.84
ScanNet++ vocabularyRaw mIoU (%)Calibrated mIoU (%)Raw absent assignment (%)Calibrated absent assignment (%)
Original (incl. D1)19.9223.73517.0321.34
+D219.7522.8618.1525.06
+D319.9323.8917.6022.77
+D2+D319.7623.2318.7226.17
LERF-OVS vocabularyRaw mIoU (%)Calibrated mIoU (%)Raw absent assignment (%)Calibrated absent assignment (%)
Original49.0965.670.000.00
+D249.0965.330.012.36
+D348.9063.590.491.81
+D2+D348.9062.430.503.99
3D-OVS vocabularyRaw mIoU (%)Calibrated mIoU (%)Raw absent assignment (%)Calibrated absent assignment (%)
Original70.7989.800.000.00
+D277.7778.1921.6639.11
+D377.7879.2621.8136.03
+D2+D377.8576.5823.0242.24

Dear Reviewer 4mbR,

With the discussion period coming to a close, we wanted to ask whether you have had a chance to review our rebuttal and revised manuscript. Following your feedback, we added full TransCLIP comparisons, an integration with Occam’s multilevel pipeline, and multiclass distractor experiments, and clarified the baseline and calibration protocols.

We would appreciate hearing whether these additions address your concerns and are happy to clarify any remaining points. If your concerns have been resolved, we would be grateful if you would consider raising your score.

Best regards, The Authors

Reviewer_5WWzOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a query-time calibration method for CLIP features lifted into 3D Gaussian scenes. The method removes the scene-specific mean direction, calibrates whitened text prototypes using a fixed generic vocabulary, and employs soft clustering and CSLS-based relevance margins for multiclass and single-query prediction, respectively. Without requiring retraining or feature re-lifting, the method achieves overall performance improvements across seven feature-lifting methods and four indoor datasets.

Critical strengths

The proposed method is highly practical, as it does not require retraining, image re-encoding, or feature re-lifting. The experiments are also relatively comprehensive, covering seven feature-lifting methods and four datasets, with overall performance improvements observed.

Critical weaknesses

The paper’s main novelty lies in the combination and application of existing components rather than the introduction of new ones. Components such as centering, whitening, soft clustering, CSLS, and generic negative phrases have all been explored in prior work. The 45-term reference vocabulary, along with some parameters and thresholds, is selected using labeled held-out indoor scenes, leaving its generalizability to new domains and encoders unclear. The single-query experiments evaluate only categories that are actually present in the scenes and do not validate the method’s ability to reject queries for absent objects. Furthermore, the paper does not report runtime or memory consumption.

Questions for authors

How well do the reference vocabulary, hyperparameters, and thresholds generalize to new datasets without recalibration using labeled data? How does the proposed method reject queries when the target object is absent from the scene? Can the authors provide separate ablation studies for scene centering and text whitening, and report the runtime and peak memory consumption during the query stage?

2 comments

We thank the reviewer for their constructive feedback. We have submitted a revised manuscript with changes marked in blue. The Common Response addresses the main concerns shared across reviewers. Below, we respond to this reviewer’s specific weaknesses and questions.

Novelty [W1]

We do not claim centering, spectral scaling, or soft clustering as new primitives. Our contribution is their formulation and evaluation as a post-hoc calibration framework for fixed lifted 3D feature fields, using a shared reference bank to support single-query and multiclass segmentation without retraining the lifted representation.

Transfer of vocabulary and parameters [W2, Q1]

We clarify our calibration protocol in our Common Response CR1.

In summary, we use the same 45 reference terms, centering strength, whitening exponent, and ridge parameter across datasets and lifting methods, and fit thresholds once on our held-out calibration scenes. Nevertheless, all our datasets are captured indoors and we have no evidence how well they transfer to outdoor scenarios. As for vision-language encoders, we have not tested other encoders and adhere to prior work, which almost exclusively uses the same encoder.

Rejection of absent queries [W3, Q2]

We thank the reviewer for pointing this out. Absent queries are an interesting aspect that we have not considered in our work. We respond to this in our Common Response CR2.

Finer grained ablations [Q3a]

We report fine-grained ablations for centering and whitening in the Common Response CR3.

Computational costs [W4, Q3b]

We report the cost of the overall query mechanism and our added calibration steps, together with their peak memory consumption in the table below. In the single-query case our one-off preparation time increases from around 0.7s to 2s and query time increases from around 4ms to 15ms, which is a significant relative increase, but would still allow for over 60 queries a second. Memory usage roughly doubles, but stays below 5GB peak. In the multi-class case our preparation takes on average around 1s and query times for raw increase to 58ms on average, while our calibration takes close to half a second on average. This increase in query time, as well as increased memory consumption up to 11.6GB, can be attributed to soft-clustering. Despite increases in query-time and memory demand we believe the calibration is well within practical bounds in most scenarios. Measurements were taken on an RTX A6000. For reference the text encoding per query takes 6.4ms on average and is additional cost shared by both raw and our calibration.

Dataset / N=1Raw prep (ms)Full prep (ms)Raw warm (ms)Full warm (ms)Raw peak VRAM (MiB)Full peak VRAM (MiB)
ScanNet101.5512.90.41.5101.3210.1
ScanNet++1069.72745.65.923.01648.03857.5
LERF-OVS511.21195.52.79.3763.41528.0
3D-OVS1371.93293.78.928.02318.54638.2
Average763.61936.94.515.51207.82558.4
Dataset / N>1Raw prep (ms)Full prep (ms)Raw warm (ms)Full warm (ms)Raw peak VRAM (MiB)Full peak VRAM (MiB)
ScanNet100.8417.52.136.8102.4510.6
ScanNet++1104.31366.6193.51166.41654.48858.8
LERF-OVS515.7808.214.7197.4764.73841.8
3D-OVS1383.11772.022.9547.02318.811631.1
Average776.01091.158.3486.91210.16210.6

Dear Reviewer 5WWz,

With the discussion period coming to a close, we wanted to ask whether you have had a chance to review our rebuttal and revised manuscript. Following your feedback, we added absent-query rejection experiments, finer-grained component ablations, and runtime and memory measurements, and clarified the scope of parameter transfer.

We would appreciate hearing whether these additions address your concerns and are happy to clarify any remaining points. If your concerns have been resolved, we would be grateful if you would consider raising your score.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Common Response (1/2)

We thank all reviewers for their constructive feedback and for recognizing the practical relevance of our method and the breadth of our evaluation. In this common response, we address three points raised by multiple reviewers:

  1. Clarification of our calibration protocol
  2. Rejection of queries for absent objects
  3. A finer-grained analysis of the contributions of our individual components

Calibration protocol [CR1]

We use the held-out scenes scene0697_02 (ScanNet) and fd361ab85f (ScanNet++) for supervised calibration choices. The reference bank and global calibration parameters are fixed before evaluation. Raw and calibrated segmentation thresholds are selected once for their respective lifting/scoring configurations and are applied unchanged to present and absent queries. Scene statistics and reference transforms are computed from each evaluation scene without semantic labels and recomputing these statistics is not label-based parameter retuning.

Rejection of absent queries [CR2]

Our original evaluation focused on present queries and did not consider queries for absent objects. In fact, parts of our calibration intuitively worsen absent query behavior, as both CSLS and relevance margin criteria might increase absent query responses. On the other hand our reference vocabulary provides an elegant way of detecting absent queries, by comparing how consistently a query's predicted Gaussian sets agree across disjoint subsets of the reference bank. In the experiment below we present absent query behavior across datasets in the single-query case for three types of distractors:

  • D1: Actual classes within the dataset that are absent from the scene's annotations (ScanNet, ScanNet++)
  • D2: Words that could commonly appear in indoor environments acting as difficult distractors: piano, carpet, guitar, violin, vacuum cleaner, treadmill, aquarium, dishwasher
  • D3: Words that typically do not appear in such environments as easy distractor: elephant, giraffe, zebra, rhinoceros, hippopotamus, crocodile, penguin, dolphin, airplane, volcano

For each dataset, we report the percentage of evaluated Gaussians predicted as foreground for an absent query (FP%), together with mIoU on present queries. As suspected, calibrating for relevance introduces a high false positive response in our calibrated method compared to just raw CLIP similarity. We saw that our calibration is very helpful for precise localization at the cost of presence detection.

Presence Gate. As a remedy we designed a very simple presence gate that assumes top responding Gaussians to be consistent when compared among disjoint reference bank subsets. Specifically, we first select the top 1% of Gaussians by raw query cosine similarity. For a subset of five reference words, we retain selected Gaussians whose query similarity exceeds the maximum similarity to those five words. We perform this comparison using nine disjoint five-word subsets that together partition our 45-word reference bank. We then average the Jaccard overlap between the resulting Gaussian sets over all 36 pairs and consider the query present if this agreement is at least 0.5. The top-1% candidate fraction and 0.5 agreement cutoff are simple heuristic choices motivated by the intended consistency test and we assume there is significant margin to improve this gating mechanism.

Results. Relative to ungated Ours, this simple gate substantially reduces absent-query foreground responses while retaining approximately 95%–97% of present-query mIoU. It does not uniformly match Raw’s lower false-positive response, but provides a practical trade-off between localization accuracy and absent-query rejection.

DatasetMethodPresent mIoUD1 FP%D2 FP%D3 FP%
ScanNetRaw26.668.501.210.34
ScanNetOurs34.529.036.503.38
ScanNetOurs + Gate33.504.320.910.13
ScanNet++Raw22.550.960.210.02
ScanNet++Ours24.393.263.161.35
ScanNet++Ours + Gate23.672.221.620.18
LERF-OVSRaw45.710.800.46
LERF-OVSOurs53.865.733.22
LERF-OVSOurs + Gate52.071.960.16
3D-OVSRaw69.211.140.46
3D-OVSOurs83.5939.2544.30
3D-OVSOurs + Gate79.614.791.46
AuthorsOpenReview ↗

Common Response 2/2

Fine-grained ablations [CR3]

Multiple reviewers asked for more details about the effects of individual components of our feature calibration as well as how the feature calibration compares to the score calibration components. To show the effects in detail we provide a more fine grained ablation study and extend it to all four datasets in the table below.

Components within Feature Calibration. The results show that scene centering (S) already improves IoU on all datasets, which is the only component that does not depend on a vocabulary choice and works solely on scene features. The vocabulary dependent components M and W use our reference vocabulary in this experiment and the results show that simple mean adjustment degrades IoU, while the spectral scaling gives substantial improvements. We have previously only considered this whitening operation in full (M+W), which performs roughly the same as just spectral scaling (W). However, while reference-mean adjustment seems hurtful when viewed in isolation, simply combining scene centering and spectral scaling (S+W), i.e. removing the potentially harmful mean adjustment from full feature calibration, fails to reach the full calbiration (S+M+W) performance on 3/4 datasets. This suggests that the interaction of the three components is non-trivial and only their combination reliably reaches best or near best results.

Feature vs. Score Calibration. The expanded ablations show that the relative contributions of feature and score calibration are dataset-dependent, rather than consistently dominated by score calibration. In isolation both improve over raw, while feature calibration alone performs better on ScanNet and LERF-OVS, whereas score calibration alone performs better on ScanNet++ and 3D-OVS. Combining them achieves the highest mIoU on all four datasets, with substantial additional gains on ScanNet and 3D-OVS, a further improvement on LERF-OVS, and performance comparable to score calibration alone on ScanNet++. These results demonstrate that feature and score calibration make complementary contributions and neither can be characterized as merely a minor refinement of the other.

Component / MC mIoU (%)ScanNetScanNet++LERF-OVS3D-OVS
Raw cosine26.8719.9249.0970.79
S (scene centering)31.1419.9959.3172.82
M (reference-mean adjustment)22.8718.0239.1069.88
W (spectral scaling)35.2520.3659.1673.51
S + M29.9919.8053.7171.83
S + W31.6519.5061.7473.11
M + W (Text Whitening)35.5720.3258.3973.10
S + M + W (Feature calibration)36.4921.1564.2373.10
Score calibration35.0923.6654.3585.88
Full calibration39.3723.7465.6789.80

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.