- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes VerdictGS, a training-free method for open-vocabulary segmentation in scenes reconstructed with Gaussian splatting. Instead of lifting multi-view CLIP features onto the Gaussians before any query, it keeps the SAM masks and CLIP features of each training view, together with their rendering correspondences to the Gaussians, and evaluates the query only when it arrives. Within each view, masks compete for the query against generic words such as "object" and "stuff", and the resulting responses are accumulated on the Gaussians through the fixed correspondences. Treating each view as a noisy annotator, the method fits a two-population binomial mixture with EM for every query and mask level. It uses the sign of the posterior log odds as a label-free decision, which can also return an empty prediction when the queried object is absent. Evidence from four mask granularities is then fused, using gated propagation from coarse to fine masks and averaging conditioned on membership. The method reaches the best LERF-OVS mIoU under the official evaluation code and is competitive on ScanNet and 3D-OVS. It also prepares scenes faster than training-free baselines, with comparable query time. The paper further proposes no-target and target accuracy for evaluating queries about absent objects. Ablations show that accumulating responses instead of features improves results under identical inputs.
Critical strengths
S1. A simple, well-controlled insight with strong results. Scoring each mask against the query within its own view before lifting, instead of lifting features and scoring afterwards, improves segmentation under identical inputs on all three benchmarks and in every LERF-OVS and ScanNet scene. The controls are convincing. Tuning the feature relevance temperature and the averaging on labels does not close the gap. Applying the same relevance to each mask without comparing it with the other masks of the view performs like feature aggregation, which isolates the cause. The full method is training-free, achieves the best LERF-OVS mIoU under the official evaluation code, and prepares scenes faster than the training-free baselines.
S2. Careful and candid evaluation. The paper separates the method's decision threshold from the benchmark's evaluation threshold, applies the official evaluation code unchanged, and marks the baselines evaluated with their own rules. It proposes a no-target protocol with N-acc and T-acc, so that the rejection of absent queries can be measured. It also reports its own failure cases, such as plain surfaces that form no target population. The EM initialization, sampling seed and execution order are specified in detail.
Critical weaknesses
W1. The per-query mixture, the paper's central idea, adds little over simpler rules. For segmentation, label-free per-query Otsu thresholding beats the mixture on both benchmarks where they are compared: 31.6 vs. 29.4 on ScanNet and 62.6 vs. 57.3 on LERF-OVS (Table 12). On LERF-OVS the mixture is also 3.5 points below the best fixed threshold, so the claim that it "segments close to the best ground-truth-selected threshold" (line 121) is generous. For absent targets, the mixture does not separate target from no-target queries better than the scores without it: AUC is 0.879 vs. 0.877 on LERF-OVS and 0.831 vs. 0.857 on ScanNet (Table 13). Its N-acc on LERF-OVS is only 35.1% (Table 14). On ScanNet, removing the mixture from the full pipeline changes mIoU only from 42.5 to 42.3 (line 1252), so most of the benchmark gains come from response accumulation and multi-level fusion rather than from the latent-class model named in the title.
W2. Fixed constants play the role of the thresholds the paper criticizes, and their sensitivity is not reported. The paper faults prior methods for thresholds selected on evaluation annotations (line 145) and claims that it "needs no threshold tuning" (line 26). Yet by Eq. (3), the winning mask of a view receives a response above one half only if its margin over the generic words exceeds . The value (line 322) and the choice of generic words therefore act as a relevance threshold. The EM initialization, the log-odds bound, the gates and the propagation pairings (Table 19) are also fixed. The paper does not say on which data they were chosen, though it notes that the method was selected among explored variants (line 505). No sensitivity analysis is given for , the generic words or the EM initialization, so it is unclear how much of the label-free behavior depends on these choices.
Questions for authors
- Sensitivity. How do mIoU, N-acc and T-acc change on the three benchmarks when is varied (for example from 50 to 200), when the generic words are replaced, and when the EM initialization changes? On which data were these values and the propagation design chosen?
- Simpler alternatives to the mixture. Would per-query Otsu for the extent, combined with a presence test on the maximum score, match the full method in both mIoU and N-acc? Does a simple per-query normalization of the response rates recover the 3D-OVS gain attributed to the mixture?
- Multiple instances and large surfaces. Responses are measured relative to the best mask in each view, so how are several instances of the queried class in one view handled, for example multiple chairs? Floor remains empty in five of the ten ScanNet scenes (line 1278). Can you report per-class results for multi-instance and surface classes?
5 comments
Response to Reviewer wuht (2/4): W2 and Q1
Response to Reviewer wuht (2/4): W2 and Q1
W2 and Q1 — "act as a relevance threshold"
W2 and Q1 — "act as a relevance threshold"
We thank the reviewer for requesting this sensitivity analysis.
The sensitivity analysis shows that the results do not depend on constants tuned to the benchmarks. and the generic words, which the reviewer identifies as a relevance threshold, were not chosen on the benchmarks. equals the logit scale that CLIP learned in training and applies to its own image–text similarities. The four generic words are LERF's canonical phrases, which seven of our baselines, including LangSplat, also use in their public code. Of the values we set, the EM initial estimates change mIoU by at most 0.6 and the log-odds bound by at most 1.8.
Sensitivity (Q1). As requested in Q1, we varied one constant at a time in the full method:
| LERF-OVS | ScanNet | 3D-OVS | |||||||
|---|---|---|---|---|---|---|---|---|---|
| Setting | mIoU | N-acc | T-acc | mIoU | N-acc | T-acc | mIoU | N-acc | T-acc |
| 67.0 | 62.2 | 95.5 | 29.7 | 88.3 | 62.5 | 95.1 | 92.7 | 87.1 | |
| 67.2 | 45.4 | 98.5 | 38.8 | 68.1 | 84.4 | 95.2 | 87.1 | 90.3 | |
| (CLIP; VerdictGS) | 68.5 | 35.1 | 100 | 42.5 | 54.3 | 91.7 | 95.3 | 79.8 | 93.5 |
| 67.2 | 26.5 | 100 | 41.6 | 33.0 | 99.0 | 95.2 | 74.2 | 93.5 | |
| 65.1 | 20.0 | 100 | 41.7 | 22.3 | 99.0 | 95.2 | 71.0 | 93.5 | |
| "object" only | 61.2 | 23.2 | 100 | 40.8 | 40.4 | 95.8 | 95.2 | 66.9 | 93.5 |
| Four other words | 66.8 | 29.7 | 100 | 39.7 | 45.7 | 93.8 | 95.2 | 72.6 | 93.5 |
| EM initial estimates | 68.4–68.5 | 34.6–36.2 | 100 | 41.9–42.5 | 48.9–55.3 | 90.6–93.8 | 95.2–95.3 | 76.6–81.5 | 90.3–96.8 |
| EM initialized from per-query Otsu | 68.4 | 35.7 | 100 | 40.1 | 51.1 | 95.8 | 88.9 | 61.3 | 93.5 |
| Log-odds bound | 66.7–68.8 | 35.1 | 100 | 42.1–42.4 | 54.3 | 91.7 | 94.8–95.3 | 79.8 | 93.5 |
Four other words: "thing", "item", "material", "pattern". EM initial estimates: 0.01–0.2, at the 90th–99th percentile, at the median or mean. Log-odds bound: , , none.
and the generic words. Every other value lowers mIoU on all three benchmarks. is the scale CLIP was trained with. A smaller flattens the responses within a view, so target masks stand out less and some target queries receive an empty prediction. A larger sharpens them, so the best-matching mask of a view receives a strong response even when the queried object is absent. The generic words are the reference that each mask's margin is measured against. Keeping only "object" can therefore only raise every margin, so more masks support absent objects and N-acc falls on all three benchmarks. Four other generic words also lower mIoU and N-acc. VerdictGS thus follows the convention that LERF established and existing methods share, and this convention is also the best of the sets tested.
Constants we set. Initializing EM from per-query Otsu instead lowers mIoU, as in our first comment, and the bound leaves N-acc and T-acc unchanged. The gates reuse , and the propagation pairings are compared in Table 19. We set these constants and the propagation design on the 19 scenes of the three benchmarks, which have no validation split.
Response to Reviewer wuht (3/4): Q3
Response to Reviewer wuht (3/4): Q3
Q3 — "several instances of the queried class in one view"; "Floor remains empty in five of the ten ScanNet scenes"
We thank the reviewer for this question.
Several instances of the queried class in one view are supported by the views in which they are the best match. Although the chair with the larger margin over the generic words receives most of the response in a view that two chairs share, almost every instance (95% on ScanNet) is the best match of its class in some of its views, and VerdictGS accumulates the responses over all views.
Chairs, the reviewer's example, are the best match in a median of only 21% of their views, yet 23 of the 30 chairs reach a vertex recall of at least 50%, among them every chair that is the best match in more than 21% of its views. Their IoU is also nearly the same in scenes with several chairs as in scenes with one (62.2 vs. 62.7). Over the eight classes that have one instance in some scenes and several in others, scenes with several instances score no lower than scenes with one (class-averaged mIoU 35.6 vs. 31.6).
Surface classes. Wall reaches 47.8 IoU and floor 34.4. Floor is lower because it receives an empty prediction in five scenes, a limitation we report in Appendix E. As the reviewer notes in S2, floor is a plain surface that forms no target population. It covers much of each scene and no view singles it out, so VerdictGS cannot tell it from an absent object. Splitting every query into two groups recovers floor but loses the empty predictions. Specifically, replacing EM with Otsu (our first comment) raises the floor IoU of scene 0070 from 0 to 61.7 but lowers the ScanNet N-acc from 54.3 to 12.8. Capturing such weakly supported targets while still returning empty predictions for absent ones is our future work (Appendix C).
We hope that these results address the reviewer's concerns.
Response to Reviewer wuht (1/4): W1 and Q2
Response to Reviewer wuht (1/4): W1 and Q2
We sincerely thank Reviewer wuht for the careful review and the constructive questions, and we are glad that the reviewer finds scoring masks within each view "a simple, well-controlled insight with strong results" and our evaluation "careful and candid". Below, we refer to the three Questions For Authors as Q1–Q3.
W1 and Q2 — "adds little over simpler rules"
We agree with the reviewer that the mixture segments no better than per-query Otsu (Table 12) and does not separate target from no-target queries better than the scores without it (AUC, Table 13). However, the mixture adds two things. First, its label-free decision boundary selects only the Gaussians whose fused evidence is positive, so VerdictGS can return an empty prediction when the queried object is absent. Second, its log odds, from which the evaluated scores are built, raise the benchmark mIoU. Neither simpler rule in Q2 achieves both on all three benchmarks with a single setting.
Empty predictions. Removing the mixture from the full pipeline lowers N-acc from 35.1–79.8 to 0 (Table 13). VerdictGS selects no threshold for any benchmark or query. Instead, EM fits the boundary to each query without labels. Per-query Otsu needs no labels either, but it always selects the higher-response group, so it never returns an empty prediction. The presence test suggested in Q2 needs a threshold selected on each benchmark's annotations, and even then the selection of the combination matches that of the full method in both mIoU and N-acc only on LERF-OVS. On ScanNet and 3D-OVS, reaching the full method's N-acc lowers the mIoU of the selected Gaussians by 1.5 and 4.5 even with the best threshold.
Some target queries also receive an empty prediction, because their responses form no target population at one or more mask levels. The 8 such queries on ScanNet account for its lower AUC. The LERF-OVS N-acc of 35.1% is obtained at T-acc 100. In contrast, feature aggregation with spatial averaging reaches 54.1% only by returning empty predictions for 8 of the 67 target queries (Table 14). At T-acc 100, it reaches at most 13.0%, even with a threshold selected on the annotations.
Benchmark mIoU. The official evaluation code computes the benchmark mIoU from the scores, not from a selection. The combination in Q2 selects on the scores of the full pipeline without the mixture, so its benchmark mIoU is that of the pipeline without the mixture. We also replaced only the EM fit with a per-query Otsu split, keeping the log odds of Eq. (5), and replaced the mixture with each normalization:
Each rule matches the mixture's mIoU on at most one benchmark: the Otsu split on LERF-OVS, and min–max on ScanNet only through floor. Normalization never returns an empty prediction, since every query keeps positive scores. It also recovers only part of the 3D-OVS gain attributed to the mixture, because the log odds also count evidence against each query (Appendix C).
W1, last sentence. As Section 4.4 states, removing the mixture lowers the benchmark mIoU on all three benchmarks: by 4.5 on LERF-OVS, 21.0 on 3D-OVS and 0.2 on ScanNet. The ScanNet change that the reviewer cites is small because the mixture gains 3.0 mIoU on the 18 classes other than floor and loses 2.8 on floor, which receives an empty prediction in five of the ten scenes. We report this limitation for plain surfaces in Appendix C, as the reviewer notes in S2. Response accumulation and multi-level fusion are also not alternatives to the latent-class model. Rather, accumulation produces the noisy annotations that it aggregates, the "views as noisy annotators" of our title, and fusion sums its per-level log odds (Eq. 7).