acceptodds
Under review as a conference paper at ICLR 2027

MonoVoc: Efficient Object-Centric Open-Vocabulary 3D Gaussian Mapping from Monocular Video

Abstract

Open-vocabulary 3D scene understanding lets users query reconstructed environments in natural language, but existing Gaussian-based methods either store dense per-Gaussian language features or couple semantics to an optimization-heavy reconstruction, both of which scale poorly on ordinary monocular video. We present MonoVoc, an object-centric framework that turns a single monocular walkthrough and its segmentation masks into a compact semantic Gaussian map paired with an object-level language database. MonoVoc attaches three learnable semantic coefficients to each Gaussian of a fixed SLAM backbone, recovers per-Gaussian semantic evidence by windowed deblending restricted to projected Gaussian footprints, and resolves discrete object identities through confidence-gated quantization that combines palette agreement with spatial consistency. Each object is then described once from a set of diverse keyframes and encoded into a single shared embedding, adding semantics on top of reconstruction without a separate language-feature distillation stage and reducing language storage from O(ND) to O(N + KD) for N Gaussians and K objects. On Replica and ScanNet, MonoVoc produces a queryable semantic map in 13–13.5 minutes end-to-end, against 186 minutes for ObjectGS, using ≈160K Gaussians and 14 MB; 84% and 91% less memory than ObjectGS and SceneSplat. Built on HI-SLAM2, MonoVoc inherits its rendering quality and reaches, 91.65 mIoU on Replica given ground-truth 2D segmentation, 84%/93% Top-1/Top-3 object retrieval, and 86% accuracy on a controlled scene-grounded QA benchmark, while ObjectGS retains an edge on object removal. A 34-participant study prefers our maps in 52.9% of responses.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D scene understanding, 3D Gaussian Splatting, Open-vocabulary, Semantic quantization, Language grounding, Object-centric mapping

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HUVpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents MonoVoc, an object-centric semantic mapping framework based on monocular Gaussian reconstruction and segmentation masks. The method augments a modified HI-SLAM2 reconstruction with compact semantic coefficients. It recovers semantic evidence through Semantic Color Deblending (SCD). Then the method assigns object identities using Confidence-Gated Quantization (CGQ), which combines palette similarity, confidence, spatial consistency, and multi-view support. Each recovered object is associated with a shared language embedding derived from representative views and object descriptions.

Rather than performing end-to-end scene understanding from RGB input alone, MonoVoc focuses on recovering, organizing, storing, and querying semantic identities within a reconstructed Gaussian map while reducing the storage overhead of dense per-Gaussian language features. Experiments on Replica and ScanNet demonstrate substantial runtime and memory reductions compared with prior semantic Gaussian methods while maintaining competitive task performance.

Critical strengths
  1. Well-motivated semantic recovery and assignment framework

The most interesting part of the paper is the semantic recovery and object-assignment pipeline. It identifies a real challenge arising from alpha compositing, where semantic evidence becomes coupled across overlapping Gaussians. Among the various steps, SCD and CGQ constitute the primary technical contributions. CGQ combines palette similarity, observation confidence, spatial consistency, and multi-view support within a unified assignment framework. This is a well-motivated design that is supported by extensive analysis in the appendix.

The language-grounding stage additionally incorporates object appearance, nearby objects, and relative spatial relationships into object descriptions. While object-level language grounding itself is not especially novel, the inclusion of spatial context is a sensible design choice that contributes to the retrieval and QA performance.

  1. Efficient semantic mapping with competitive downstream performance

One of the key observations is that object-level semantic representations can significantly reduce storage and computation compared with dense per-Gaussian language-feature approaches while retaining much of their functionality. The reported runtime, memory footprint, retrieval performance, object removal capability suggest that substantial compression is possible without completely sacrificing downstream utility.

That said, these efficiency gains should be interpreted in light of differing assumptions across methods, including the use of ground-truth segmentation supervision and different reconstruction pipelines.

Critical weaknesses
  1. Reliance on ground-truth segmentation

The biggest concern I have is that the entire semantic pipeline depends on ground-truth segmentation masks. As described in the paper, ground-truth segmentation is used for supervision and evaluation. Consequently, the semantic metrics primarily measure how well semantic labels are preserved and recovered within the representation rather than end-to-end semantic understanding in the conventional sense. This simplifies the problem. The practical robustness of the pipeline under realistic segmentation errors remains unclear and is likely one of the most important factors determining real-world usefulness.

This also complicates comparisons to baselines such as ObjectGS and SceneSplat. These methods make different assumptions regarding semantic supervision, representation learning, and semantic feature generation. As a result, the reported runtime, memory, and semantic-performance comparisons are not strictly conducted under the same settings.

  1. Moderate novelty

The area of object-centric and language-enabled scene representations is very crowded, including ObjectGS, SceneSplat, and others. As a result, the novelty of object-level language storage, object retrieval, and language grounding is relatively limited. The paper is best characterized as a systems paper whose strongest technical ideas are SCD and CGQ.

Questions for authors
  1. Why is learning semantic coefficients followed by SCD preferable to a simpler pipeline such as segmentation masks → Gaussian association → majority-vote labeling? This would materially clarify the contribution of SCD.

  2. CGQ combines palette distinctiveness, vote agreement, and support consistency into a single confidence score. Can the authors provide an ablation isolating the contribution of each individual component? This would help determine whether all components are necessary.

  3. The paper emphasizes lightweight semantic mapping, yet all experiments are performed on an NVIDIA A100 80GB GPU. What computational resources are required for the recommended configuration?

Reviewer_G9J6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

MonoVoc builds an object-centric, language-queryable 3D Gaussian map from a monocular video and 2D GT instance masks. Building on a modified HI-SLAM2 map, it learns three semantic coefficients per Gaussian, applies an adaptive semantic palette (ASPT), footprint-restricted semantic colour deblending (EWR/SCD), and confidence-gated quantization to assign object IDs. Gemini then describes selected object views, and LLaMA encodes a shared language representation per object.

Critical strengths

The experimental section of the paper is comprehensive, covering validation experiments for various tasks along with the corresponding analysis of the results.

Critical weaknesses
  • Strong Reliance on Ground-Truth Instance Masks. The proposed method requires 2D ground-truth instance masks with strict cross-view consistency, which is unrealistic in practical scenarios. Most existing 3DGS-based OV scene understanding methods (OpenGaussian, InstanceGaussian, PanoGS,...) rely on noisy 2D instance masks generated by segmentation models. This strong assumption raises concerns about the method's robustness and limits its scalability and applicability to real-world scenes.

  • Questionable Segmentation Processing Strategy. The use of ASPT to distinguish segmentation masks based on their colors is conceptually questionable, as these colors are arbitrary and serve only visualization purposes. The meaningful information lies in the discrete instance labels rather than their RGB values. Given the availability of ground-truth instance masks, 3D Gaussians could be segmented more directly through projection and rasterization using instance labels. Therefore, the motivation and necessity of ASPT are unclear, and this design deviates from common practices in existing research.

Questions for authors
  • How does the performance of segmentation, retrieval, and question answering (QA) change when using predicted 2D instance masks, e.g. from SAM?

  • Does the runtime reported in Figure 3 include 2D mask generation, Gemini-based description generation, and LLaMA encoding/inference?

  • With the HI-SLAM2 geometry, masks, and number of Gaussians fixed, how do removing SCD/ASPT, replacing confidence-gated quantization with nearest-palette assignment, isolating the confidence factor, and using simpler text descriptions affect the final Gaussian-rendered mIoU, retrieval, and QA performance?

Reviewer_jk2QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MonoVoc, a method that transforms monocular video and its corresponding 2D segmentation masks into an object-centric 3D Gaussian map, which supports natural language queries. Built upon HI-SLAM2, the approach augments each Gaussian with three semantic coefficients. It employs a palette transformation to enhance the separability of identity-related colors and utilizes weighted aggregation within local windows to recover semantic color evidence, subsequently assigning object identities by integrating confidence scores with spatial consistency. Furthermore, the method generates textual descriptions for each object from multiple viewpoints and stores shared language embeddings, enabling object retrieval, question answering, and editing. Experiments conducted on the Replica and ScanNet datasets evaluate reconstruction quality, label mapping, downstream interaction capabilities, and computational efficiency. As the primary experiments utilize ground-truth 2D segmentation labels, the focus is on the propagation of known 2D object information into a 3D map, rather than the autonomous discovery of previously unknown objects from raw video.

Critical strengths
  1. The system’s objectives are clearly oriented toward practical applications. By integrating monocular reconstruction, object identity association, and language-based querying into a unified pipeline—enabling tasks such as retrieval, question answering, and object removal—the approach demonstrates tangible utility.
  2. Object-level language storage offers a distinct efficiency advantage. Sharing language embeddings across Gaussian primitives representing the same object avoids redundant storage of high-dimensional features, reducing the storage complexity for language-associated data from O(ND)O(ND) to O(N+KD)O(N+KD). Furthermore, the use of local windows and batch processing provides a concrete mechanism for managing the computational overhead of semantic aggregation.
  3. The paper provides a detailed algorithmic description and localized analysis. The appendix includes experiments on quantization, color palettes, and regularization configurations, as well as an explanation of the optimization process for Semantic-Consistent Decomposition (SCD) and an error analysis. The authors explicitly acknowledge that SCD serves as an approximate aggregation rather than an exact inverse solution in the general case, and they also disclose limitations stemming from discrepancies in segmentation supervision and input noise.
Critical weaknesses
  1. The applicability of the method to autonomous object discovery under real-world input conditions remains unverified. Section 4.1 relies on ground-truth 2D segmentation labels; the current experiments do not demonstrate robustness against missed detections, instance merging, identity switching, or the emergence of previously unsegmented objects. The analysis of mask noise provided in the appendix serves merely as a proposed diagnostic framework rather than an empirical experiment.
  2. The novel contributions of key modules relative to existing literature require clearer substantiation. The organization of object-level features and descriptions is already present in ConceptGraphs, while the normalized weighted aggregation used in SCD closely resembles the 2D feature-lifting mechanism found in LUDVIG (ICCV 2025). Although the manuscript cites these works, it lacks a direct analysis of algorithmic differences and comparisons under equivalent conditions. Furthermore, since low-dimensional identity features can also facilitate object-level language sharing, the potential for storage savings does not, in itself, establish the necessity or superiority of a three-channel semantic color representation.
  3. While the theoretical analysis supports the validity of approximate estimation, it is insufficient to guarantee reliable semantic unmixing or error correction. Appendix M demonstrates the optimality of SCD for specific local targets; however, this optimal solution does not necessarily correspond to the true semantic coefficients that generated the observations. Overlaps and transparency continue to introduce bias, and erroneous labels that remain consistent across frames may erroneously attain high confidence scores.
  4. The ablation studies fail to adequately isolate the contributions of individual strategies within the overall system. While the experiments concerning color palettes and quantization configurations are extensive, the appendix does not isolate specific factors—such as color intervals, cross-frame voting, and observation-based confidence weighting—to assess their individual impact. The specific advantages of SCD over simple label voting or existing weighted-lifting methods, the benefits of three-channel encoding versus compact identity features, and the independent contributions of multi-view selection and description generation remain insufficiently validated. Additionally, some quantization-related ablations utilize point-based rendering, whereas the primary experiments employ Gaussian-based rendering; it remains to be shown whether the localized gains observed in the former reliably translate to the main task.
  5. Discrepancies in supervision, language modules, and resource scales across the compared methods complicate the attribution of the reported results. This paper utilizes 2D masks from different sources compared to some of the baselines; furthermore, MonoVoc’s multi-view description generation and text encoding pipeline differ from approaches that rely directly on vision-language feature matching. Consequently, the performance gap cannot be attributed solely to the 3D semantic mapping method. Moreover, a map storage requirement of approximately 14 MB is not equivalent to the VRAM required for runtime execution; the actual experiments were conducted using an A100 80GB GPU. To draw valid conclusions regarding direct deployment and efficiency, it is necessary to clarify whether the complete pipeline accounts for the costs associated with mask generation, language model inference, and other external processing steps.
Questions for authors
  1. Does the current system include a complete pipeline for automatically generating 2D instance masks and establishing cross-frame identity associations? Are there empirical results using these predicted masks that demonstrate the system's sensitivity to missed detections, identity switches, and instance merging? Please distinguish between components that have already been validated and those that are proposed for future implementation.
  2. What are the specific algorithmic contributions of SCD, distinct from the normalized rendering-weight aggregation used in LUDVIG? Are there directly comparable results available, obtained under identical geometric, masking, and observational conditions?
  3. Are there ablation results showing the impact of removing color-based interval filtering, cross-frame voting, and observation-based support, respectively? Do the existing experiments demonstrate that the benefits of confidence-based gating are independent of the choices made regarding spatial smoothing and distance metrics?
  4. Keeping the set of object candidates, 2D supervision, and language generation/encoding modules constant, which existing comparative results support the advantages of MonoVoc in retrieval or question-answering tasks? How are the respective contributions of 3D mapping and the language model disentangled?
  5. Please provide a detailed breakdown of the reported total runtime, map storage requirements, and GPU memory usage—specifically accounting for 2D mask generation, cross-frame identity association, Gemini API calls, and LLaMA encoding. When transitioning to a new video, which steps require re-optimization or re-computation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.