acceptodds
Withdrawn from ICLR 2027

GaussAsset: Evidence-Grounded Segmentation into Au- ditable Object Assets in Indoor 3D Gaussian Scenes

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D Gaussian Splatting provides an explicit rendering representation for indoor scenes, but turning that representation into objects remains difficult: real captures contain floaters and missing coverage, and exhibit view-dependent effects such as transparency and reflections. Methods built for clean point clouds or surface annotations do not account for these Gaussian-specific artifacts and do not decide which Gaussians belong to which object, while fixed-view lifting methods do not establish persistent ownership. To address this problem, we present GaussAsset, an evidence-grounded met hod that turns a Gaussian carrier into auditable object assets by separating semantic proposal from ownership commitment. A frozen vision- language model proposes scene concepts and flags uncertain relations; a geometry- constrained executor lifts SAM3 masks through rendered depth, associates them across views, and commits mutually exclusive ownership over source Gaussians only when the evidence supports it, recording every decision in an auditable ledger. To evaluate this setting, we introduce GaussAsset-100, a benchmark of 100 furnished scenes with 3,590 human-verified object cuboids. The annotations target countable, exportable entities above a size gate and are provided as object- level cuboids, the unit downstream users export; membership diagnostics are box-derived and explicitly marked. Under a controlled shared-observation protocol that fixes the inputs for all methods, GaussAsset achieves the strongest object recall and all-GT bbox mIoU among the compared methods (77.5/67.2 at IoU 0.25/0.50; 55.4 bbox mIoU) and 56.1 scene-macro semantic mIoU without target- domain training. Together, these results support separating semantic proposal from ownership commitment as a practical route from a Gaussian carrier to auditable, exportable object assets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, 3D instance segmentation, active perception, open-vocabulary segmentation, vision-language models, 3D scene understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VTX8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents GaussAsset, a training-free framework aimed at decomposing reconstructed 3D Gaussian Splatting (3DGS) indoor scenes into individual object assets. The methodology separates semantic proposal generation (using a frozen vision-language model and SAM3) from Gaussian ownership assignment. The spatial assignment pipeline projects rendered 2D masks into 3D using depth maps, applies bird’s-eye-view (BEV) and plane-based geometric constraints, associates proposals across viewpoints, and commits points to instances via a margin-based ownership gate (Aik(1)≥τA,Aik(1)−Aik(2)≥τmarginA_{ik}(1) \ge \tau_A, A_{ik}(1) - A_{ik}(2) \ge \tau_{\text{margin}}). The paper also introduces GaussAsset-100, an indoor benchmark of 100 scenes containing 3,590 human-verified oriented bounding boxes. Under a static shared-observation protocol, the paper evaluates 3D bounding box detection metrics and box-derived membership proxies against two clustering baselines (MaskClustering, SAI3D) and one semantic baseline (Chorus).

Critical strengths
  1. Annotating 3,590 oriented 3D bounding boxes across 100 multi-room indoor Gaussian scenes represents a non-trivial engineering effort in benchmark building, providing structured bounding annotations for scenes with multi-view reconstructive noise.

  2. The paper presents a thoroughly documented multi-stage pipeline that integrates BEV-based floor splitting, camera visibility pruning, and multi-view mask association into a practical, training-free workflow.

Critical weaknesses
  1. The paper exhibits a fundamental disconnect between its central theoretical claims and its empirical evaluation protocol. The core thesis posits that the framework solves the fine-grained problem of determining primitive-level Gaussian ownership, specifically addressing floater geometry, view-dependent effects, transparent boundaries, and ambiguous scene contacts. However, the ground-truth annotations in the introduced GaussAsset-100 benchmark consist strictly of oriented 3D bounding boxes (cuboids) without any human-verified point-level or Gaussian-level segmentation labels. All reported primitive-level diagnostics—including purity, welding, and fragmentation—are computed using spatial bounding box inclusion as an automated proxy. In realistic, furnished indoor environments, 3D bounding boxes inevitably intersect in common configurations, such as chairs pushed beneath dining tables, keyboards placed on desks, or decor placed inside open shelving units. Within these overlapping volumes, defining ground truth purely by box inclusion introduces severe ground-truth label contamination and artificial welding by construction. As a direct result, the only direct, non-proxy evaluation metrics in Table 2 are bounding box mIoU and bounding box recall at 0.25 and 0.50 IoU thresholds. The evaluation therefore reduces an ambitious primitive-level Gaussian disentanglement framework to a coarse 3D bounding box detection task, providing no rigorous evidence that physical Gaussian boundaries or object surfaces are accurately decoupled.

  2. The comparative evaluation avoids established state-of-the-art baselines in 3D Gaussian Splatting scene parsing. While semantic and instance segmentation on 3DGS has seen substantial development in recent literature, the baseline selection in this work is exceptionally sparse. In Table 2, the proposed framework is compared exclusively against MaskClustering (a minimal heuristic clustering baseline) and SAI3D (a point-cloud approach adapted to Gaussians). Widely recognized, peer-reviewed 3DGS instance and semantic segmentation methods that specifically address Gaussian feature lifting, affinity learning, and boundary grouping—such as Gaussian Grouping, SAGA, Feature 3DGS, and OpenGaussian—are entirely absent from the benchmark. Furthermore, the semantic segmentation evaluation in Table 3 evaluates against only a single baseline, Chorus. By omitting the most prominent and competitive methods in the 3DGS segmentation literature, the empirical superiority and practical utility of GaussAsset remain unsubstantiated.

  3. The ablation analysis exhibits a severe attribution fallacy regarding where performance improvements actually originate. The proposed pipeline stacks multiple high-capacity off-the-shelf components, including SAM3, a frozen vision-language model, bird's-eye-view ground plane filtering, heuristic clipping contracts, and the proposed ownership arbitration mechanism. In Section 5.3, the ablation progression shows that mAP begins at 5.6 for geometry-only processing, jumps by 8.3 points to 13.9 upon introducing the VLM vocabulary prompt, and reaches 16.6 with basic BEV floor splitting and cleaning heuristics, before reaching 20.6 for the reference configuration. This progression reveals that the vast majority of performance gains are driven by the semantic priors of the 2D foundation models and basic scene pre-filtering heuristics, rather than the core mathematical formulations emphasized in the narrative, such as the auditable ledger or the margin-based ownership equations. Furthermore, the ownership ablation in Table 5 justifies the ownership gate by comparing it against a naive same-label union baseline that yields an extreme welding rate of 29.64%. Comparing against an unconstrained merger of all scene primitives sharing the same class label across disconnected spatial regions constitutes an artificial strawman, as standard segmentation systems do not merge spatially disparate clusters without geometric affinity constraints.

  4. There is a pronounced narrative divergence between the conceptual positioning of the system and its actual experimental execution. The introduction and method sections place heavy emphasis on an active perception architecture featuring a shared VLM controller, risk-routed verification, and closed-loop dynamic viewpoint planning. However, Sections 5.1 and 5.2 explicitly reveal that the mainline benchmark results are generated entirely under a static, pre-rendered shared observation protocol with predetermined camera poses. The active perception mechanism is relegated to a minor, non-essential pilot study that does not contribute to the quantitative results in Table 2 or Table 3. Similarly, framing the auditable ledger as a mechanism to resolve Gaussian-specific rendering defects, such as floaters and view-dependent photometric variations, is conceptually flawed. Recording provenance rays, view histories, and ownership decisions in an audit log provides retrospective traceability, but it lacks any physical, mathematical, or radiometric mechanism to eliminate non-physical floaters or resolve multi-view rendering inconsistencies.

The experimental validation is largely confined to an internally curated benchmark without verification on established public 3D datasets. The empirical claims rely almost entirely on GaussAsset-100, an environment where the scene selection, annotation guidelines, size-gate thresholds, and proxy evaluation metrics were all established internally. Standard, community-standardized indoor benchmarks for 3D instance and semantic understanding—such as ScanNet200, Replica, or ScanNet++—are omitted from the main body of the paper. While brief transfer evaluations on InteriorGS and LERF-OVS are noted, they are deferred to the supplementary material without detailed comparative analysis against competitive methods. Evaluating primarily on an author-designed benchmark with author-defined proxy metrics severely limits the external validity, reproducibility, and generalizability of the reported conclusions.

Questions for authors
  1. Given that GaussAsset-100 relies on 3D oriented bounding boxes rather than human-verified primitive-level or mesh ground truth, how does the evaluation protocol handle Gaussians located inside overlapping bounding box volumes, such as chairs nested beneath desks or objects positioned on tables? Are primitives in these shared intersection zones assigned to multiple classes, ignored, or arbitrarily assigned to one bounding box? How can the authors demonstrate that the reported 1.7% welding rate reflects genuine separation of physical surface geometry rather than an artifact of how proxy labels are resolved in intersecting box regions?

  2. What specific methodological or practical factors led to the exclusion of prominent 3D Gaussian segmentation frameworks, such as Gaussian Grouping, SAGA, and OpenGaussian, from the comparisons in Table 2 and Table 3? Could the authors provide a direct quantitative comparison against these established baselines under standardized mean Average Precision (mAP) metrics on public benchmark datasets, such as ScanNet200 or Replica?

  3. Beyond serving as a structured storage log for tracking decision history and provenance metadata, does the auditable ledger contribute functionally to the optimization, continuous regularization, or geometric refinement of the 3D Gaussian primitives? If the ledger mechanism is reduced to a standard ephemeral array and discarded immediately after inference, does the final segmentation accuracy or scene parsing quality degrade in any measurable way?

  4. In Section 5.3, the introduction of the 2D foundation model vocabulary accounts for more than half of the total cumulative mAP improvement (5.6 to 13.9). What quantitative performance is achieved if the identical frozen VLM and SAM3 2D proposals are lifted via depth back-projection and clustered using standard spatial density clustering (such as DBSCAN or HDBSCAN) combined with connected component analysis, under the same geometric filtering rules? What is the exact isolated margin of improvement provided specifically by the STAR-3D ownership gate over this conventional clustering baseline?

  5. In the mainline experiments reported in Table 2 and Table 3, what exact proportion of the 3,590 evaluated instances had their segmentation boundaries influenced or adjusted by the active, risk-routed view selection mechanism? If the mainline benchmark was evaluated entirely under a static camera trajectory, how do the authors justify presenting active perception and closed-loop verification as primary architectural contributions in the paper's title, abstract, and introduction?

  6. The introduction asserts that the framework directly addresses Gaussian-specific artifacts, including floater primitives, incomplete coverage, transparency, and specular reflections. Which specific mathematical formulation or geometric operation in the pipeline actively suppresses or removes non-physical floaters and resolves view-dependent appearance ambiguities, beyond standard heuristic confidence thresholding and camera visibility clipping?

Reviewer_1mUVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GaussAsset, a training-free pipeline that decomposes an existing indoor 3DGS scene into mutually exclusive object assets over the original Gaussians. A frozen Qwen-family VLM proposes a scene vocabulary and assists with scene cleaning and floor splitting. SAM3 masks are lifted to Gaussians through rendered depth and associated across views. Each Gaussian is assigned to an instance only if its support and its margin over the second-best instance pass thresholds; otherwise it remains unassigned. All decisions are logged in an “ownership ledger.” The paper also introduces GaussAsset-100, a benchmark of 100 real furnished-showroom 3DGS scenes with 3,590 human-verified oriented cuboids. Validation consists of a comparison with MaskClustering and SAI3D (recall 77.5/67.2 at IoU 0.25/0.50, bbox mIoU 55.4), a semantic comparison with Chorus, ownership-rule ablations, a three-scene observation-policy pilot, and transfer results on InteriorGS and LERF-OVS.

Critical strengths
  • The paper provides a benchmark of real 3DGS captures with human-verified object-level cuboids, which is a relatively scarce resource for evaluating object extraction from Gaussian scenes.
  • The ownership-rule ablations (Tables 12–14) are controlled and come with paired bootstrap statistics. They show that identity consolidation and margin-based gating reduce welding. However, neither strength outweighs the weaknesses below.
Critical weaknesses
  1. Limited methodological novelty; a key line of prior work is ignored. The core lifting and assignment step closely follows Splat Feature Solver (ICLR 2026) and its predecessors, including the framing. Eq. 20 averages mask contributions weighted by per-view blending weights. This is the row-sum weighted-average lifting rule that Splat Feature Solver formalizes as a linear inverse problem, and which CosegGaussians, Occam’s LGS and Dr.Splat also used. The object-assignment step, an exclusive assignment with an unassigned set, is already addressed by the clustering in Splat Feature Solver, which assigns each splat a cluster ID or a −1 “unclassified” label, and by its predecessor LAGA (ICML 2025). None of these works is cited or compared. Once this line of work is considered, the remaining contribution is mainly replacing SAM with SAM3 and adding a VLM for vocabulary proposal, which I do not consider a fundamental contribution.
  2. The dataset contribution is limited. A comparable benchmark can be built by training 3DGS on ScanNet++ scenes and transferring the existing instance annotations to the Gaussians and to bounding boxes. Much of this has already been done, e.g., SceneSplat-7K provides 3DGS reconstructions of ScanNet++ with transferred labels. Such a benchmark would also provide real per-Gaussian membership labels, whereas GaussAsset-100 offers only cuboids, so all ownership metrics in the paper depend on box-inclusion proxies.
Questions for authors
  • (Clarification) Beyond replacing the foundation models with SAM3 and a Qwen VLM, what is technically different from Splat Feature Solver, LAGA, Occam’s LGS and Dr.Splat? Please compare against Splat Feature Solver and LAGA on GaussAsset-100 using the same SAM3 masks. If the gap is small, my assessment of the method contribution stands.
  • (Clarification) What does GaussAsset-100 provide that cannot be obtained from ScanNet++ 3DGS reconstructions with transferred instance labels (e.g., SceneSplat-7K)? If you claim capture-artifact differences, please quantify them.
  • (Clarification) Thresholds are said to be selected on a validation set, but all 100 scenes are used for evaluation. Which scenes were used for tuning?
Reviewer_cKP5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The submission proposes GaussAsset, which decomposes an existing indoor 3D Gaussian Splatting scene into object assets without modifying the underlying reconstruction. A frozen VLM proposes scene vocabulary, SAM3 generates masks from rendered views, and a geometric backend lifts and associates these masks, before assigning each Gaussian to at most one object. Unsupported primitives remain unassigned, while an ownership ledger record the evidence behind assignments. The work also introduces GaussAsset-100, a dataset of 100 real scenes with 3590 human-verified object cuboids. Experiment report improved object recall and bounding-box mIoU under shared observations, and semantic segmentation results.

Critical strengths
  1. The submission addresses a useful problem beyond object localization.

During editing, a bounding box alone does not specify which Gaussian primitives should be exported or removed. It is a meaningful practical objective that produce source-indexed object subsets with explicit handling of ambiguous and excluded primitives.

  1. The geometric ownership rules are motivated.

Depth consistency, cross-view support, and ownership margins address recognizable problems such as merging adjacent furniture or assigning occluded geometry to the wrong object. Allowing uncertain primitives to remain unassigned is preferable to forcing a complete but unreliable decomposition.

  1. The shared-observation comparison is a good experimental choice.

Controlling the views and mask front end help to isolate the 3D parsing backend. Relative to SAI3D, the reported gains in [email protected] and bbox mIoU, together with lower welding, are convincing. The ownership ablations also show that naive same-label merging is inadequate.

  1. The real-scene benchmark could be a useful contribution.

The dataset preserves reconstruction artifacts rather than relying only on clean synthetic geometry. The submission also explicitly distinguishes human cuboid annotations from box-derived membership proxies, which is important for expalining the results.

Critical weaknesses
  1. The novelty over reconstruction followed by grounding and instance fusion is not sufficiently established.

    The evaluated pipeline follows a familiar structure: render a reconstructed scene, then obtain 2D masks, then lift them through depth, and then associate them into 3D instances. The distinction is therefore not “grounding versus ownership,” since lifting-based instance methods can also produce persistent 3D memberships. The contribution appear to lie in the particular association schedule, conservative assignment rules, and provenance tracking. These may be useful, but the submission need a sharper comparison against the closest training-free Gaussian methods, particularly FlashSplat and Lifting by Gaussians, rather than relying primarily on MaskClustering and SAI3D.

  2. The annotations do not directly validate the central ownership claim.

    Cuboids support evaluation of object localization and extent, but they do not establish whether an exported Gaussian subset is clean and complete. Box inclusion can assign background or neighboring geometry to an object, especially near contact boundaries. So, the reported purity and welding numbers remain proxy measurements of the property the submission most emphasizes. A small independent membership audit or an exported-asset evaluation would substantially strengthen the evidence.

  3. The presentation gives active verification more prominence than the experiments support.

    The main figures feature ambiguity-driven view requests and subsequent verification. However, the full closed-loop procedure is reserved, while the view-selection pilot covers only three scenes and introduce no additional VLM calls. The authors discuss this limitation, but the presentation still make it difficult to separate the evaluated method from the proposed extension. The main results support a fixed-observation parsing backend, not yet a demonstrated active verification system.

  4. Several experimental definitions and configurations are difficult to reconcile.

    In Appendix B.4, it describes an assignment gate that uses GT-dependent matching, which includes a degenerate case in which predictions are assigned to the smallest GT box. Its relationship to the inference-time ownership gate is unclear, making the coverage results hard to interpret. The production reference in Tab. 15 also differs from the production rule in Tab. 5 and Tab. 12, without a clear configuration mapping. Moreover, removing identity consolidation improves PQ and point F1 despite worsening welding, so the results support a trade-off rather than an unqualified improvement.

Questions for authors
  1. What is the main algorithmic contribution beyond existing mask-lifting and instance-association pipelines?

Can the authors isolate this contribution experimentally, and compare against the closest training-free Gaussian methods? If a shared-mask comparison is infeasible, a separate same-carrier evaluation should still be informative.

  1. Can the authors provide an ownership evaluation independent of cuboid inclusion?

For example, manually auditing a subset of exported objects for contamination and missing geometry would help to establish whether better box metric translate into better assets.

  1. Please specify the membership evaluator precisely.

I'm a bit curious that how are predictions assigned to GT objects, how are coverage and welding computed, and how are primitives outside annotated boxes treated? Which gates operate during inference, and which are used only for evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.