acceptodds
Under review as a conference paper at ICLR 2027

VerdictGS: Views as Noisy Annotators for Query-Time Open-Vocabulary Gaussian Splatting Segmentation

Abstract

Existing methods for open-vocabulary segmentation with 3D Gaussian Splatting integrate multi-view language features into 3D representations before any query. This adds aggregation, registration, and often grouping costs; moreover, query-independent aggregation mixes observations before their relevance to the query is known, predefined groups constrain the target's extent, and thresholds must be tuned before the query yet still ignore query-specific score distributions. We propose VerdictGS, a query-time inference framework that builds a 3D relevance field directly from individual observations' responses to a text query. Treating each view as a noisy annotator, VerdictGS aggregates per-view text–mask responses onto Gaussians and fits a per-query two-population binomial mixture to them with EM and without labels, yielding membership evidence whose decision boundary adapts to the query; this evidence is then fused across mask granularities. Because every decision is made at query time, VerdictGS builds no language representation before the query, lets each query weigh the original observations rather than their mixture, determines the target extent per Gaussian rather than within predefined groups, and needs no threshold tuning. Reducing this inference to sparse matrix operations and low-dimensional model fits, VerdictGS prepares scenes faster than training-free baselines and answers queries in comparable time, while achieving strong segmentation accuracy under each benchmark's evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-vocabulary 3D segmentation, Gaussian splatting, training-free, query-time inference, label aggregation, expectation-maximization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wuhtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VerdictGS, a training-free method for open-vocabulary segmentation in scenes reconstructed with Gaussian splatting. Instead of lifting multi-view CLIP features onto the Gaussians before any query, it keeps the SAM masks and CLIP features of each training view, together with their rendering correspondences to the Gaussians, and evaluates the query only when it arrives. Within each view, masks compete for the query against generic words such as "object" and "stuff", and the resulting responses are accumulated on the Gaussians through the fixed correspondences. Treating each view as a noisy annotator, the method fits a two-population binomial mixture with EM for every query and mask level. It uses the sign of the posterior log odds as a label-free decision, which can also return an empty prediction when the queried object is absent. Evidence from four mask granularities is then fused, using gated propagation from coarse to fine masks and averaging conditioned on membership. The method reaches the best LERF-OVS mIoU under the official evaluation code and is competitive on ScanNet and 3D-OVS. It also prepares scenes faster than training-free baselines, with comparable query time. The paper further proposes no-target and target accuracy for evaluating queries about absent objects. Ablations show that accumulating responses instead of features improves results under identical inputs.

Critical strengths

S1. A simple, well-controlled insight with strong results. Scoring each mask against the query within its own view before lifting, instead of lifting features and scoring afterwards, improves segmentation under identical inputs on all three benchmarks and in every LERF-OVS and ScanNet scene. The controls are convincing. Tuning the feature relevance temperature and the averaging on labels does not close the gap. Applying the same relevance to each mask without comparing it with the other masks of the view performs like feature aggregation, which isolates the cause. The full method is training-free, achieves the best LERF-OVS mIoU under the official evaluation code, and prepares scenes faster than the training-free baselines.

S2. Careful and candid evaluation. The paper separates the method's decision threshold from the benchmark's evaluation threshold, applies the official evaluation code unchanged, and marks the baselines evaluated with their own rules. It proposes a no-target protocol with N-acc and T-acc, so that the rejection of absent queries can be measured. It also reports its own failure cases, such as plain surfaces that form no target population. The EM initialization, sampling seed and execution order are specified in detail.

Critical weaknesses

W1. The per-query mixture, the paper's central idea, adds little over simpler rules. For segmentation, label-free per-query Otsu thresholding beats the mixture on both benchmarks where they are compared: 31.6 vs. 29.4 on ScanNet and 62.6 vs. 57.3 on LERF-OVS (Table 12). On LERF-OVS the mixture is also 3.5 points below the best fixed threshold, so the claim that it "segments close to the best ground-truth-selected threshold" (line 121) is generous. For absent targets, the mixture does not separate target from no-target queries better than the scores without it: AUC is 0.879 vs. 0.877 on LERF-OVS and 0.831 vs. 0.857 on ScanNet (Table 13). Its N-acc on LERF-OVS is only 35.1% (Table 14). On ScanNet, removing the mixture from the full pipeline changes mIoU only from 42.5 to 42.3 (line 1252), so most of the benchmark gains come from response accumulation and multi-level fusion rather than from the latent-class model named in the title.

W2. Fixed constants play the role of the thresholds the paper criticizes, and their sensitivity is not reported. The paper faults prior methods for thresholds selected on evaluation annotations (line 145) and claims that it "needs no threshold tuning" (line 26). Yet by Eq. (3), the winning mask of a view receives a response above one half only if its margin over the generic words exceeds log⁡(Mv)/β\log(M_v)/\beta. The value β=100\beta = 100 (line 322) and the choice of generic words therefore act as a relevance threshold. The EM initialization, the log-odds bound, the gates and the propagation pairings (Table 19) are also fixed. The paper does not say on which data they were chosen, though it notes that the method was selected among explored variants (line 505). No sensitivity analysis is given for β\beta, the generic words or the EM initialization, so it is unclear how much of the label-free behavior depends on these choices.

Questions for authors
  1. Sensitivity. How do mIoU, N-acc and T-acc change on the three benchmarks when β\beta is varied (for example from 50 to 200), when the generic words are replaced, and when the EM initialization changes? On which data were these values and the propagation design chosen?
  2. Simpler alternatives to the mixture. Would per-query Otsu for the extent, combined with a presence test on the maximum score, match the full method in both mIoU and N-acc? Does a simple per-query normalization of the response rates recover the 3D-OVS gain attributed to the mixture?
  3. Multiple instances and large surfaces. Responses are measured relative to the best mask in each view, so how are several instances of the queried class in one view handled, for example multiple chairs? Floor remains empty in five of the ten ScanNet scenes (line 1278). Can you report per-class results for multi-instance and surface classes?
5 comments
AuthorsOpenReview ↗

Response to Reviewer wuht (1/4): W1 and Q2

Response to Reviewer wuht (1/4): W1 and Q2

We sincerely thank Reviewer wuht for the careful review and the constructive questions, and we are glad that the reviewer finds scoring masks within each view "a simple, well-controlled insight with strong results" and our evaluation "careful and candid". Below, we refer to the three Questions For Authors as Q1–Q3.

W1 and Q2 — "adds little over simpler rules"

We agree with the reviewer that the mixture segments no better than per-query Otsu (Table 12) and does not separate target from no-target queries better than the scores without it (AUC, Table 13). However, the mixture adds two things. First, its label-free decision boundary selects only the Gaussians whose fused evidence is positive, so VerdictGS can return an empty prediction when the queried object is absent. Second, its log odds, from which the evaluated scores are built, raise the benchmark mIoU. Neither simpler rule in Q2 achieves both on all three benchmarks with a single setting.

Empty predictions. Removing the mixture from the full pipeline lowers N-acc from 35.1–79.8 to 0 (Table 13). VerdictGS selects no threshold for any benchmark or query. Instead, EM fits the boundary to each query without labels. Per-query Otsu needs no labels either, but it always selects the higher-response group, so it never returns an empty prediction. The presence test suggested in Q2 needs a threshold selected on each benchmark's annotations, and even then the selection of the combination matches that of the full method in both mIoU and N-acc only on LERF-OVS. On ScanNet and 3D-OVS, reaching the full method's N-acc lowers the mIoU of the selected Gaussians by 1.5 and 4.5 even with the best threshold.

Some target queries also receive an empty prediction, because their responses form no target population at one or more mask levels. The 8 such queries on ScanNet account for its lower AUC. The LERF-OVS N-acc of 35.1% is obtained at T-acc 100. In contrast, feature aggregation with spatial averaging reaches 54.1% only by returning empty predictions for 8 of the 67 target queries (Table 14). At T-acc 100, it reaches at most 13.0%, even with a threshold selected on the annotations.

Benchmark mIoU. The official evaluation code computes the benchmark mIoU from the scores, not from a selection. The combination in Q2 selects on the scores of the full pipeline without the mixture, so its benchmark mIoU is that of the pipeline without the mixture. We also replaced only the EM fit with a per-query Otsu split, keeping the log odds of Eq. (5), and replaced the mixture with each normalization:

ConfigurationLERF-OVS mIoUScanNet mIoU3D-OVS mIoU
VerdictGS68.542.595.3
Without the mixture (Q2)64.042.374.3
Otsu split instead of EM68.841.489.0
Min–max normalization63.943.583.9
Z-score normalization64.137.188.3

Each rule matches the mixture's mIoU on at most one benchmark: the Otsu split on LERF-OVS, and min–max on ScanNet only through floor. Normalization never returns an empty prediction, since every query keeps positive scores. It also recovers only part of the 3D-OVS gain attributed to the mixture, because the log odds also count evidence against each query (Appendix C).

W1, last sentence. As Section 4.4 states, removing the mixture lowers the benchmark mIoU on all three benchmarks: by 4.5 on LERF-OVS, 21.0 on 3D-OVS and 0.2 on ScanNet. The ScanNet change that the reviewer cites is small because the mixture gains 3.0 mIoU on the 18 classes other than floor and loses 2.8 on floor, which receives an empty prediction in five of the ten scenes. We report this limitation for plain surfaces in Appendix C, as the reviewer notes in S2. Response accumulation and multi-level fusion are also not alternatives to the latent-class model. Rather, accumulation produces the noisy annotations that it aggregates, the "views as noisy annotators" of our title, and fusion sums its per-level log odds (Eq. 7).

AuthorsOpenReview ↗

Response to Reviewer wuht (2/4): W2 and Q1

Response to Reviewer wuht (2/4): W2 and Q1

W2 and Q1 — "act as a relevance threshold"

W2 and Q1 — "act as a relevance threshold"

We thank the reviewer for requesting this sensitivity analysis.

The sensitivity analysis shows that the results do not depend on constants tuned to the benchmarks. β\beta and the generic words, which the reviewer identifies as a relevance threshold, were not chosen on the benchmarks. β=100\beta = 100 equals the logit scale that CLIP learned in training and applies to its own image–text similarities. The four generic words are LERF's canonical phrases, which seven of our baselines, including LangSplat, also use in their public code. Of the values we set, the EM initial estimates change mIoU by at most 0.6 and the log-odds bound by at most 1.8.

Sensitivity (Q1). As requested in Q1, we varied one constant at a time in the full method:

LERF-OVSScanNet3D-OVS
SettingmIoUN-accT-accmIoUN-accT-accmIoUN-accT-acc
β=50\beta = 5067.062.295.529.788.362.595.192.787.1
β=75\beta = 7567.245.498.538.868.184.495.287.190.3
β=100\beta = 100 (CLIP; VerdictGS)68.535.110042.554.391.795.379.893.5
β=150\beta = 15067.226.510041.633.099.095.274.293.5
β=200\beta = 20065.120.010041.722.399.095.271.093.5
"object" only61.223.210040.840.495.895.266.993.5
Four other words66.829.710039.745.793.895.272.693.5
EM initial estimates68.4–68.534.6–36.210041.9–42.548.9–55.390.6–93.895.2–95.376.6–81.590.3–96.8
EM initialized from per-query Otsu68.435.710040.151.195.888.961.393.5
Log-odds bound66.7–68.835.110042.1–42.454.391.794.8–95.379.893.5

Four other words: "thing", "item", "material", "pattern". EM initial estimates: π\pi 0.01–0.2, p1p_1 at the 90th–99th percentile, p0p_0 at the median or mean. Log-odds bound: ±25\pm 25, ±100\pm 100, none.

β\beta and the generic words. Every other value lowers mIoU on all three benchmarks. β=100\beta = 100 is the scale CLIP was trained with. A smaller β\beta flattens the responses within a view, so target masks stand out less and some target queries receive an empty prediction. A larger β\beta sharpens them, so the best-matching mask of a view receives a strong response even when the queried object is absent. The generic words are the reference that each mask's margin is measured against. Keeping only "object" can therefore only raise every margin, so more masks support absent objects and N-acc falls on all three benchmarks. Four other generic words also lower mIoU and N-acc. VerdictGS thus follows the convention that LERF established and existing methods share, and this convention is also the best of the sets tested.

Constants we set. Initializing EM from per-query Otsu instead lowers mIoU, as in our first comment, and the bound leaves N-acc and T-acc unchanged. The gates reuse β\beta, and the propagation pairings are compared in Table 19. We set these constants and the propagation design on the 19 scenes of the three benchmarks, which have no validation split.

AuthorsOpenReview ↗

Response to Reviewer wuht (3/4): Q3

Response to Reviewer wuht (3/4): Q3

Q3 — "several instances of the queried class in one view"; "Floor remains empty in five of the ten ScanNet scenes"

We thank the reviewer for this question.

Several instances of the queried class in one view are supported by the views in which they are the best match. Although the chair with the larger margin over the generic words receives most of the response in a view that two chairs share, almost every instance (95% on ScanNet) is the best match of its class in some of its views, and VerdictGS accumulates the responses over all views.

Chairs, the reviewer's example, are the best match in a median of only 21% of their views, yet 23 of the 30 chairs reach a vertex recall of at least 50%, among them every chair that is the best match in more than 21% of its views. Their IoU is also nearly the same in scenes with several chairs as in scenes with one (62.2 vs. 62.7). Over the eight classes that have one instance in some scenes and several in others, scenes with several instances score no lower than scenes with one (class-averaged mIoU 35.6 vs. 31.6).

Surface classes. Wall reaches 47.8 IoU and floor 34.4. Floor is lower because it receives an empty prediction in five scenes, a limitation we report in Appendix E. As the reviewer notes in S2, floor is a plain surface that forms no target population. It covers much of each scene and no view singles it out, so VerdictGS cannot tell it from an absent object. Splitting every query into two groups recovers floor but loses the empty predictions. Specifically, replacing EM with Otsu (our first comment) raises the floor IoU of scene 0070 from 0 to 61.7 but lowers the ScanNet N-acc from 54.3 to 12.8. Capturing such weakly supported targets while still returning empty predictions for absent ones is our future work (Appendix C).

We hope that these results address the reviewer's concerns.

Reviewer_uT22OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents VerdictGS, a training-free method for open-vocabulary segmentation in 2D Gaussian Splatting scenes that performs language reasoning at query time. It caches SAM masks, CLIP features, and mask-to-Gaussian correspondences, then aggregates query-dependent mask responses across views using soft voting and a two-population binomial mixture model. The resulting posterior enables label-free target selection, including rejection of absent targets. The method further integrates multiple SAM granularities through gated cross-level fusion.

Critical strengths

View-wise scoring is well-motivated and supported by controlled experiments.

  • The paper identifies a limitation of feature lifting, i.e. aggregating multi-view features onto Gaussians loses the ability to compare regions within each view and adaptively weight observations. VerdictGS addresses this by scoring masks before lifting them to Gaussians. Controlled comparisons show consistent improvements over feature aggregation across all scenes.

Adaptive target selection and absent-target evaluation address an important limitation.

  • The per-query mixture model provides an adaptive, label-free decision rule, avoiding reliance on fixed or test-tuned thresholds. It also enables empty predictions when queried objects are absent, a capability lacking in many baselines. The proposed N-acc/T-acc protocol highlights this underexplored evaluation setting, while the distinction between decision and evaluation thresholds improves methodological clarity.
Critical weaknesses

Query-time computation and memory scalability remain unestablished.

  • The efficiency analysis relies on a single small scene with 299 views (Table 3, Appendix H). VerdictGS reprocesses cached masks across all views and four granularities for each query, with additional sparse mask-to-Gaussian aggregation passes. Therefore, query latency and memory usage may grow substantially with scene size, unlike feature-lifting baselines whose query costs are largely independent of the number of views. The 10510^5-Gaussian subsampling limits only the mixture-model fitting cost, not the aggregation overhead. Without query-time and memory measurements across increasing view counts and larger scenes, the reported efficiency may not generalize to room- or building-scale mapping or repeated interactive queries.

The effectiveness of the label-free decision rule and absent-target detection is insufficiently demonstrated.

  • The main benchmark results rely on evaluation-specific thresholds or argmax decisions, so they primarily assess the relevance field rather than the proposed mixture-based verdict. When evaluated directly (Table 12), the mixture underperforms per-query Otsu thresholding and the best fixed threshold on LERF-OVS (57.3 versus 62.6 and 60.8) and Otsu on ScanNet (29.4 versus 31.6).
  • Absent-target detection is also limited: LERF-OVS N-acc is only 35.1%, below the 54.1% of feature aggregation with spatial averaging, while target/no-target ranking shows no improvement on LERF-OVS and declines on other datasets. Empty predictions additionally depend on the mixture initialization, and large, weakly distinctive targets can be missed.
Questions for authors
  • How do query latency and memory scale with the number of views (e.g., subsampling 50 → 1000+ views) and with scene size? Can you evaluate on larger multi-room scenes and report the footprint of AkA^k and the caches?
  • Are there strategies (view subsampling, caching per-view responses, amortizing across query batches) that keep query cost bounded as captures grow?
  • Can you report N-acc/T-acc for the main baselines under their own selection rules, and explain why the mixture lowers AUC on ScanNet and 3D-OVS?
7 comments
AuthorsOpenReview ↗

Response to Reviewer uT22 (1/5): W1 and Q1

Response to Reviewer uT22 (1/5): W1 and Q1

We sincerely thank Reviewer uT22 for the careful review and the constructive questions, and we are glad that the reviewer finds view-wise scoring "well-motivated" and sees absent-target evaluation as addressing "an important limitation".

We refer to the two Critical Weaknesses as W1 and W2, to the two bullets of W2 as its first and second points, and to the Questions For Authors as Q1–Q3.

W1 and Q1 — "query latency and memory usage may grow substantially with scene size, unlike feature-lifting baselines whose query costs are largely independent of the number of views"; Q1: "Can you evaluate on larger multi-room scenes …?"

We measured how the query cost grows with scene size and with the number of views.

Our query cost does not grow substantially with scene size. As in feature lifting, it grows only with the number of Gaussians, and on 10 ScanNet scenes joined into one (1.57 M Gaussians, 1,158 views), a query takes 12 ms with text encoding and 0.68 GiB. Over 1,000 views of one area, it takes 15 ms and 1.0 GiB when each Gaussian keeps at most 50 views, which leaves the accuracy on our benchmarks almost unchanged.

Feature lifting merges all views into one feature per Gaussian before the query. In contrast, we keep an AkA^k entry for each view that sees a Gaussian, so that the query can score every view. A query therefore compares the text with every mask and reads one entry per Gaussian and observing view, so its cost scales mainly with the number of AkA^k entries, the Gaussians times the views per Gaussian. A larger scene adds Gaussians but not views per Gaussian, and a dense capture needs no more than 50 views per Gaussian (Q2). Across 79 scene setups, query time and memory follow the number of AkA^k entries (correlation 0.97 and 0.99) rather than the number of views (0.51 and 0.44).

The reference below is feature lifting with a CLIP feature on every Gaussian. Occam's LGS and VALA, however, keep features on fewer Gaussians, so we compare only how the costs grow.

OursReference
SceneViewsGaussiansEntries per GaussianQuery (ms)Memory (GiB)Query (ms)Memory (GiB)
scene0000_002790.08 M1941.80.110.70.47
10 ScanNet scenes1,1581.57 M836.80.6816.68.98
figurines2990.93 M2357.70.8211.05.32
figurines1,1960.93 M94027.73.1211.15.32

Query times exclude text encoding (5.3 ms).

Larger scene (first two rows, Q1's multi-room scene). From one ScanNet scene to ten joined into one, the Gaussians grow from 0.08 M to 1.57 M, but each is still seen only by the views of its own scene. Per Gaussian, the joined scene takes 0.46 KB, within the 0.2–1.4 KB of the 19 single scenes, so our memory grows in proportion to the Gaussians, as the reference's does at 6 KB per Gaussian. The same argument applies at building scale, which is larger than any scene we measured.

More views of the same area (last two rows). No benchmark scene has more than 299 views, so for Q1's 1000+ we repeat our densest capture four times. These views see Gaussians that the 299 views already cover, and 50 views per Gaussian are enough, because keeping at most 50 leaves the accuracy almost unchanged (Q2). A query over the 1,196 views then takes 15 ms with text encoding and 1.0 GiB, against 13 ms and 0.82 GiB for the 299 views. For comparison, Occam's LGS takes 10 ms and VALA 5.8 ms in Table 3.

Hence the efficiency of Table 3 holds beyond one small scene. On the joined scene, and on dense captures with at most 50 views per Gaussian, repeated interactive queries take 12–15 ms each, text encoding included.

AuthorsOpenReview ↗

Response to Reviewer uT22 (2/5): W1 (continued), Q1 and Q2

Response to Reviewer uT22 (2/5): W1 (continued), Q1 and Q2

W1 (continued) — "The 10510^5-Gaussian subsampling limits only the mixture-model fitting cost, not the aggregation overhead"

All query times in W1 already include the aggregation overhead over all Gaussians. For example, on the joined scene, a query over its 1.57 M Gaussians takes 6.8 ms. As the reviewer notes, the 10510^5 sampled Gaussians limit only the mixture-model fitting cost, and this saving is small, since fitting EM on all 1.24 M observed Gaussians raises the query from 6.8 to 8.0 ms.

Q1 (continued) — "report the footprint of AkA^k and the caches"

Like the query time, the memory does not grow substantially. It is 0.68 GiB for the 1.57 M Gaussians of the joined scene, and 0.99 GiB for 1,196 views of one area with at most 50 views per Gaussian.

Most of it is AkA^k. On the joined scene, AkA^k takes 0.44 GiB, the mask features 0.15 GiB and the caches of Appendix B.5 0.09 GiB. AkA^k keeps one entry of about 4 bytes for each view and level in which a Gaussian is seen. Per Gaussian, the joined scene keeps 0.46 KB, against 0.23 KB for its 3DGS parameters. Even the 0.99 GiB of the densest capture stays within the 0.01–3.4 GiB that the baselines of Table 3 keep for their language representations on figurines.

Q2 — "Are there strategies (view subsampling, caching per-view responses, amortizing across query batches) that keep query cost bounded as captures grow?"

Two of the three suggested strategies, view subsampling and batching, apply to VerdictGS and reduce the query cost with almost no change in accuracy. The third, caching per-view responses, does not apply, because a response depends on the query.

View subsampling. We subsample per Gaussian rather than per capture. Specifically, the response rate of each Gaussian is computed from its KK views of largest rendering weight, while ngn_g in Eq. (5) still counts all of its views. Gaussians seen in KK views or fewer keep all of theirs, so only densely captured areas change.

Query (ms)Memory (GiB)LERF-OVSScanNet3D-OVS
Views per Gaussian299 views1,196 views1,196 viewsmIoUT-accmIoUmIoU
All7.727.73.1268.010042.595.2
K=50K = 505.69.90.9968.010042.595.2
K=20K = 204.06.50.6465.910042.695.2

Query time and memory on figurines.

At K=50K = 50, mIoU changes by at most 0.05 and N-acc by at most 1.1 on the three benchmarks, and T-acc is unchanged. We therefore offer K=50K = 50 as a budget for dense captures, while using every view by default.

Caching per-view responses. A response compares the masks of a view for the given query (Eq. 3), so it cannot be computed before the query. This comparison is the view-wise scoring that the reviewer finds "well-motivated", and it gives the gain over feature aggregation (Table 4). Everything that does not depend on the query is already computed once per scene (Appendix B.5).

Amortizing across query batches. Batched queries share each pass over AkA^k, so on the joined scene the time per query falls from 6.8 to 4.8 ms.

Hence view subsampling per Gaussian and batching reduce the query cost as captures grow, without removing the per-view responses that raise mIoU over feature aggregation (Table 4). The revision states in Section 4.3 that the remaining cost is a limitation for captures denser than those we measured.

We hope these results address W1, Q1 and Q2.

AuthorsOpenReview ↗

Response to Reviewer uT22 (3/5): W2 (first point and N-acc) and Q3 (baselines)

Response to Reviewer uT22 (3/5): W2 (first point and N-acc) and Q3 (baselines)

W2, first point — "primarily assess the relevance field rather than the proposed mixture-based verdict"; "underperforms per-query Otsu thresholding and the best fixed threshold"

As the reviewer notes, the benchmarks assess the relevance field rather than the verdict. This field, however, is built from the mixture's log odds, so the benchmark mIoU also reflects the mixture. Removing the mixture lowers it from 68.5, 42.5 and 95.3 to 64.0, 42.3 and 74.3 on LERF-OVS, ScanNet and 3D-OVS (Appendix E).

Table 12 evaluates the verdict directly, by the mIoU of its selection and by N-acc and T-acc. There, as the reviewer notes, per-query Otsu and, on LERF-OVS, the best fixed threshold segment the present targets better. The best fixed threshold, however, is chosen on each benchmark's ground truth, as in several existing methods (Section 2), and it does not transfer between benchmarks. For instance, τ = 0.1, the best on LERF-OVS, gives N-acc 13.8% on ScanNet, against 56.4% for the mixture. VerdictGS selects no threshold for any benchmark or query. Instead, EM fits the boundary to each query without labels. Per-query Otsu needs no labels either, but it never returns an empty prediction (N-acc 0).

Decision ruleLabelsmIoUN-accT-acc
ScanNet
Per-query mixturenone29.456.488.5
Per-query Otsunone31.60.0100.0
Fixed threshold τ = 0.1GT-selected27.513.8100.0
Fixed threshold τ = 0.15GT-selected29.118.199.0
LERF-OVS
Per-query mixturenone57.341.6100.0
Per-query Otsunone62.60.0100.0
Fixed threshold τ = 0.1GT-selected60.840.598.5
Fixed threshold τ = 0.15GT-selected58.651.497.0

The benchmark results therefore already reflect the mixture, through the field it builds, and its verdict uses no threshold chosen on labels.

W2, second point, and Q3 — "LERF-OVS N-acc is only 35.1%, below the 54.1% of feature aggregation"; Q3: "N-acc/T-acc for the main baselines under their own selection rules"

The 54.1% comes with a lower T-acc. On LERF-OVS at the same T-acc of 100%, VerdictGS has a higher N-acc than feature aggregation and than every baseline we evaluated under its own selection rule.

A rule can raise N-acc by also emptying target queries. Feature aggregation reaches 54.1% by emptying 8 of the 67 target queries (Table 4(a)), and at T-acc 100% even a threshold chosen on the ground truth gives it at most 13.0%.

Most baselines cannot return an empty prediction at all, "a capability lacking in many baselines" as the reviewer notes. Their released code selects relative to each query's maximum, so the top region always passes (N-acc 0, T-acc 100%).

MethodsSelection ruleEmpty prediction
LangSplatV2, Occam's LGS, VALA, THGS, LEGORelative to each query's maximumNever
Dr. Splat, LaGaFixed thresholdPossible
Splat Feature SolverFrom the number of annotated objectsUndefined for a no-target query
All methods of Table 2 (ScanNet)Highest-scoring class per vertexNo per-query selection

Of the two baselines with a fixed threshold, only Dr. Splat has released evaluation code. Run on the same queries with its own threshold, 0.55, it reaches N-acc 30.3% at T-acc 100%.

At T-acc 100%, 35.1% is therefore higher than the N-acc of feature aggregation and of every baseline we evaluated under its own selection rule, and VerdictGS reaches it without a test-tuned threshold.

As our Limitations and Conclusion state, the benchmarks cannot yet score empty predictions. Scoring each method's own decision, as N-acc and T-acc do, is therefore a natural next step for 3D benchmarks.

Reviewer_VAL7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents VerdictGS, an inference-time framework for open-vocabulary 3D segmentation on 2D Gaussian Splatting representations. Instead of pre-baking heavy multi-view vision-language embeddings onto Gaussians, VerdictGS stores only sparse 2D-to-3D rendering correspondences offline. At query time, text-mask agreement is evaluated within each view against null tokens, treating views as noisy annotators within a Dawid–Skene-inspired latent-class framework. A two-population binomial mixture is fitted via unsupervised EM across SAM mask levels, using the sign of the log posterior odds as an adaptive decision boundary. The method demonstrates competitive results on standard benchmarks like LERF-OVS and ScanNet, with fast scene preparation and the ability to handle absent targets.

Critical strengths
  1. Treating multi-view 2D observations as noisy annotators and inferring 3D target membership through an unsupervised binomial mixture is a creative and principled alternative to conventional heuristic feature-lifting pipelines.
  2. Evaluating query-mask agreement competitively within each viewpoint before 3D lifting effectively mitigates premature semantic dilution and avoids irreversible mixing of object and context features.
  3. Offline preparation is very lightweight. Avoiding per-scene neural field training and feature quantization allows scene preparation to complete in under a minute.
Critical weaknesses
  1. Parameter collapse on broad, uniform regions: As discussed in the paper, the binomial mixture can struggle on large, visually uniform structures like floors or large single-colored furniture. When relative contrast against generic tokens is low across views, the fitted parameters risk collapsing (p1≈p0p_1 \approx p_0), resulting in false-negative empty masks.
  2. Linear runtime scaling with query vocabulary (O(Q)O(Q)): Because the mixture fitting and sparse matrix traversals run at query time, querying QQ candidate categories requires QQ passes. For dense scene parsing over larger vocabularies, this per-query overhead could become a practical bottleneck compared to feature fields evaluated via batched matrix multiplication.
Questions for authors
  1. Could coarse geometric priors (such as surface normals or height coordinates) be easily incorporated into the mixture formulation to prevent parameter collapse on large structural surfaces like floors?
  2. How does latency scale if querying a larger set of classes concurrently, and can the EM iterations and sparse graph operations be effectively batched across queries on the GPU?
  3. How robust is the fitted threshold for fringe Gaussians that are visible from only one or two viewpoints (e.g., ng≤2n_g \le 2)?
3 comments
AuthorsOpenReview ↗

Response to Reviewer VAL7 (1/3): W1 and Q1

Response to Reviewer VAL7 (1/3): W1 and Q1

We thank the reviewers for their careful reviews. Below, we describe the implementation behind the timing numbers in our responses and list the changes in the revised PDF.

Optimized implementation. All timing and memory numbers in our responses and in the revised Table 3 come from an optimized implementation of the same method: preparation in one GPU pass, AkA^k stored once in float16 (4 instead of 16 bytes per entry), each query run as one CUDA graph, and pruning of AkA^k entries whose alpha-blending weight, summed over the mask, is below 0.01. mIoU and mAcc differ from the submission by at most 0.031, and all accuracy numbers in the paper remain those of the submitted implementation. Timed end to end as the baselines of Table 3, including loading and saving, on the same GPU and 3DGS reconstruction of figurines, it prepares the scene in 18.7 s (baselines: 61–547 s) and answers a query in about 13 ms, text encoding included.

Changes in the revision (uploaded, with changes in blue):

  • Query cost (uT22, VAL7). Section 4.3 and Appendix H report how query time and memory grow with the number of Gaussians and with more views of the same area, how a 50-view budget and batched classes reduce the cost, and which costs remain limitations. Section 4.1 states that the 10510^5 sampled Gaussians serve only the EM fit.
  • What the mixture adds (uT22, wuht). Section 4.4 and Table 5 compare the mixture with per-query Otsu and fixed thresholds in mIoU, N-acc and T-acc. We replace "segments close to" with the measured gap to the best fixed threshold on each benchmark, and the introduction, contributions and conclusion state instead that the mixture's boundary can return empty predictions for absent objects and its log odds raise the benchmark mIoU. Section 4.4 also explains why removing the mixture lowers the ScanNet mIoU least.
  • Empty predictions and missed targets (uT22, wuht, VAL7). Sections 4.2 and 4.4 compare N-acc on LERF-OVS at T-acc 100% with feature aggregation and with Dr. Splat under its own threshold, and explain the lower AUC. The Limitations now discuss the missed broad surfaces, Appendix C adds where geometry can enter the model, and Appendix D adds several instances of a class in one view.
  • Gaussians seen in few views (VAL7). Appendix B.2 shows how the fitted threshold rises as the number of observing views falls.
  • Constants (wuht, uT22). The abstract states that VerdictGS selects no threshold for any benchmark or query. Section 4.1 gives where β\beta and the generic words come from, Table 6 in Section 4.4 gives the sensitivity to the constants we set, and Table 7 in Appendix B.1 gives the results with the same constants on 3DGS reconstructions.
  • Table numbers. The reviews and our responses use the submitted numbering. The revision adds Tables 6 and 7 and merges the two 3D-OVS tables into Table 8, so submitted Tables 8–19 become Tables 9–20 (e.g., Table 12 → 13), and Table 21 replaces Table 20.

This list will be revised and extended during the discussion.

AuthorsOpenReview ↗

Response to Reviewer VAL7 (2/3): W2, Q2 and Q3

Response to Reviewer VAL7 (2/3): W2, Q2 and Q3

W2 and Q2 — "querying QQ candidate categories requires QQ passes … this per-query overhead could become a practical bottleneck compared to feature fields evaluated via batched matrix multiplication"; Q2: "How does latency scale if querying a larger set of classes concurrently, and can the EM iterations and sparse graph operations be effectively batched across queries on the GPU?"

VerdictGS evaluates each view at query time, so it builds no language representation before the query and segments more accurately than feature aggregation. For the same reason, as the reviewer notes, its query time grows linearly with the number of classes. The EM iterations and the sparse graph operations, however, can be batched across classes on the GPU, so querying all 19 ScanNet classes on a scene takes 7.7 ms, and querying 1,000 classes takes 0.49 s.

With no language representation built before the query, VerdictGS prepares figurines in 18.7 s, against 61–547 s for the baselines of Table 3 (general response). Comparing the masks within each view against the query raises the ScanNet mIoU from 22.7 with feature aggregation to 35.3 under identical inputs (Table 4). This comparison depends on the query, so it is repeated for each class.

These per-class computations can be batched on the GPU (Q2). Because each class is inferred independently of the others (Sec. 4.1), a batch of classes runs as a single CUDA graph. Within this graph, the sparse graph operations become sparse matrix–matrix products that read AkA^k once for the whole batch, and each EM iteration updates the mixtures of all classes in the same kernel launches, with separate parameters for each class. Compared with querying classes one at a time, batching lowers the query time per class by 18–73% without changing the sign of any Gaussian's score. Each class still adds its own EM fit and its own column to the sparse products, so the query time still grows linearly with the number of classes, but at this lower cost per class:

scene0000_00figurines
ClassesOursFeature fieldOursFeature field
19 (ScanNet classes)7.71.0——
100594.464065
1,000490405,900580

Time (ms) to query all classes in one batch, without text encoding. Feature field: a CLIP feature on every Gaussian at three levels, scored with LERF's relevance.

When classes are queried in a batch, a feature field is faster per class, because it only multiplies precomputed features. Lowering the per-class cost of query-time evaluation while keeping its accuracy is therefore the next step of this work. The revision states this limitation in Section 4.3.

Q3 — "How robust is the fitted threshold for fringe Gaussians that are visible from only one or two viewpoints?"

The fitted threshold is robust for fringe Gaussians. It rises as the number of observing views falls, so Gaussians seen in one or two views rarely make the prediction of a no-target query non-empty.

Eq. (B4) computes this threshold from a Gaussian's number of observing views ngn_g and from the mixture parameters. These parameters are fitted on all Gaussians of the level, not on the fringe Gaussian's own one or two views. In Eq. (5), the log odds are the sum of a prior term and one term per observing view. The prior term is negative, because the target is a minority in almost every fit. A Gaussian seen in few views therefore has little evidence to outweigh this prior, and it needs a higher response rate to obtain positive log odds. In the fitted mixtures, the median threshold is 0.95 for a Gaussian seen in one view and only 0.18 for a Gaussian seen in twenty views. A Gaussian seen in one view therefore typically needs almost full support from that view to be selected. As a result, on all three benchmarks, fringe Gaussians are less frequent among the Gaussians selected for no-target queries than among all observed Gaussians. Only one of the 403 no-target queries has a non-empty prediction made of fringe Gaussians alone. The score of a fringe Gaussian also depends on more than its own views. Its response rate is the regional average of Eq. (B2) over the masks that contain it, and its final score is averaged again within its medium masks (Sec. 3.4).

We hope these answers address the reviewer's concerns.

AuthorsOpenReview ↗

Response to Reviewer VAL7 (3/3): Changes in the revision

Response to Reviewer VAL7 (3/3): Changes in the revision

Changes in the revision. The changes for this review are listed, marked "VAL7", under "Changes in the revision" in our general response.

We thank the reviewer again and would be glad to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

General Response

We thank the reviewers for their careful reviews. Below, we describe the implementation behind the timing numbers in our responses and list the changes in the revised PDF.

Optimized implementation. All timing and memory numbers in our responses and in the revised Table 3 come from an optimized implementation of the same method: preparation in one GPU pass, AkA^k stored once in float16 (4 instead of 16 bytes per entry), each query run as one CUDA graph, and pruning of AkA^k entries whose alpha-blending weight, summed over the mask, is below 0.01. mIoU and mAcc differ from the submission by at most 0.031, and all accuracy numbers in the paper remain those of the submitted implementation. Timed end to end as the baselines of Table 3, including loading and saving, on the same GPU and 3DGS reconstruction of figurines, it prepares the scene in 18.7 s (baselines: 61–547 s) and answers a query in about 13 ms, text encoding included.

Changes in the revision (uploaded, with changes in blue):

  • Query cost (uT22, VAL7). Section 4.3 and Appendix H report how query time and memory grow with the number of Gaussians and with more views of the same area, how a 50-view budget and batched classes reduce the cost, and which costs remain limitations. Section 4.1 states that the 10510^5 sampled Gaussians serve only the EM fit.
  • What the mixture adds (uT22, wuht). Section 4.4 and Table 5 compare the mixture with per-query Otsu and fixed thresholds in mIoU, N-acc and T-acc. We replace "segments close to" with the measured gap to the best fixed threshold on each benchmark, and the introduction, contributions and conclusion state instead that the mixture's boundary can return empty predictions for absent objects and its log odds raise the benchmark mIoU. Section 4.4 also explains why removing the mixture lowers the ScanNet mIoU least.
  • Empty predictions and missed targets (uT22, wuht, VAL7). Sections 4.2 and 4.4 compare N-acc on LERF-OVS at T-acc 100% with feature aggregation and with Dr. Splat under its own threshold, and explain the lower AUC. The Limitations now discuss the missed broad surfaces; Appendix C adds where geometry can enter the model, and Appendix D adds several instances of a class in one view.
  • Gaussians seen in few views (VAL7). Appendix B.2 shows how the fitted threshold rises as the number of observing views falls.
  • Constants (wuht, uT22). The abstract states that VerdictGS selects no threshold for any benchmark or query. Section 4.1 gives where β\beta and the generic words come from, Table 6 in Section 4.4 gives the sensitivity to the constants we set, and Table 7 in Appendix B.1 gives the results with the same constants on 3DGS reconstructions.
  • Table numbers. The reviews and our responses use the submitted numbering. The revision adds Tables 6 and 7 and merges the two 3D-OVS tables into Table 8, so submitted Tables 8–19 become Tables 9–20 (e.g., Table 12 → 13), and Table 21 replaces Table 20.

This list will be revised and extended during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.