acceptodds
Under review as a conference paper at ICLR 2027

ActiveLang: Active Open-Vocabulary 3D Mapping with Semantic-Uncertainty-Guided Exploration

Abstract

As robots increasingly assist humans with diverse tasks, they need both geometric and semantic understanding of their surroundings. Moreover, robots often operate in unfamiliar environments and take on new tasks without knowing the relevant concepts ahead of time. This motivates language-annotated 3D maps that support open-vocabulary scene understanding and human–robot interaction. We introduce ActiveLang, an autonomous system for active open-vocabulary 3D mapping with semantic-uncertainty-guided exploration. ActiveLang performs online language-feature adaptation on a compact dual-Gaussian representation to jointly reconstruct scene geometry, appearance, and open-vocabulary semantics with modest memory overhead. Its planner efficiently selects informative viewpoints, enabling effective mapping with fewer observations and lower computational cost. Experiments on Replica and ScanNet++ demonstrate substantial improvements in 2D and 3D open-vocabulary segmentation over both online and offline baselines, highlighting that actively exploring scenes builds language-annotated 3D maps more efficiently.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
active semantic mapping, 3D Gaussian Splatting, open-vocabulary mapping, active exploration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZBf1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ActiveLang is an active open-vocabulary 3D mapping system given RGB-D observations and ground-truth camera poses. It uses dual Gaussian branches with shared centers for geometry/appearance and language features, online feature adaptation with a prototype bank, and a next-view planner combining geometric coverage, semantic uncertainty, and travel cost. The paper evaluates 2D view segmentation and localization, 3D point-cloud segmentation, and reconstruction on Replica and ScanNet++.

Critical strengths
  • This paper brings online language-feature adaptation, dense 3D mapping, and autonomous view selection into one loop. Its class-agnostic prototype mechanism addresses imbalanced semantic observations

  • The paper provides a comprehensive experimental evaluation covering 2D and 3D semantic understanding, as well as 3D reconstruction performance.

Critical weaknesses
  • Limited Novelty. The paper primarily appears to be a system-level integration of active reconstruction and open-vocabulary semantic understanding. The active reconstruction component builds upon prior work (ActiveGAMER + SplaTAM), while the semantic representation and compression modules are also largely derived from existing methods (OnlineLangSplat). Overall, the contribution seems incremental, with limited methodological novelty beyond combining existing techniques.

  • The main and Appendix 2D results conflict. For ScanNet++, Table 2 reports ActiveLang 2D SLAM/NVS mIoU of 21.01/20.13, whereas averaging the eight scenes in Table A4, explicitly presented as its per-scene breakdown, gives approximately 30.86/30.67. All four 2D columns for all three methods disagree, while the 3D columns agree. The reported results are inconsistent across different tables, raising serious concerns about their reliability and credibility. These inconsistencies make the experimental findings difficult to trust and represent a significant issue that needs to be addressed.

  • Limited Effectiveness of the Proposed Modules. The ablation results in Table 4 suggest that the proposed modules do not consistently improve performance. In several cases, removing certain modules actually leads to better or even the best results, raising concerns about their effectiveness and necessity.

  • Limited Ablation Evaluation. Why are the ablation studies conducted on only four scenes from the Replica dataset instead of all eight? Given the previously mentioned inconsistencies in the reported results, I would also like to see the ablation results on all eight ScanNet++ scenes to further verify the reliability and consistency of the proposed method.

Questions for authors
  • Please verify and correct the ScanNet++ 2D results reported in Tables 2 and A4, and explain the reasons for the inconsistencies.

  • Although the paper claims lower computational costs for online reconstruction, it does not provide a fair and comprehensive comparison of end-to-end runtime with existing methods to substantiate this claim.

  • Concerns about Evaluation Fairness. Do active exploration and SLAM reconstruction start from the same camera position? Figure A3 (Office2 and Room1) suggests that their starting positions may differ. Additionally, do both methods use the same number of frames for reconstruction? Differences in the number of input frames could compromise the fairness of the comparison. Furthermore, the blue test split appears to contain only virtual camera viewpoints oriented toward a single central point, which may not adequately evaluate novel view synthesis performance.

4 comments
AuthorsOpenReview ↗

Response to R3(ZBf1)

Thank you for raising these questions.

W1. Limited Novelty

Please refer to our response to R2(PT1s) W1.

W2 / Q1. The main and Appendix 2D results conflict.

Thank you for catching this error in the caption. The ScanNet++ 2D results in Table 2 evaluate all classes, whereas Table A4 uses the top-10 most frequent classes per scene; the 3D results are top-10 classes in both tables. We apologize that the captions did not distinguish these settings. We've submitted a revised version with corrected captions and evaluation descriptions, top-10 averages, and per-scene all-class results added to the appendix. The changes are highlighted in blue.

  • We attached the ScanNet++ results averaged over eight scenes here, using the top-10 most frequent ground-truth categories per scene for all 2D and 3D semantic metrics, ActiveLang outperforms both baselines under both settings.
MethodAE trained on same framesAE type2D SLAM mIoU ↑2D SLAM Loc ↑2D NVS mIoU ↑2D NVS Loc ↑3D mIoU ↑3D mAcc ↑Chamfer (m) ↓
LangSplat×Offline12.3932.9413.3336.1812.2223.241.611
OnlineLangSplatting✓Online13.6146.9413.6943.6616.2423.711.444
ActiveLang (Ours)✓Online30.8655.2430.7550.8821.4830.491.278

W3. Limited Effectiveness of the Proposed Modules

  • Active semantic mapping involves multiple objectives—semantic accuracy, geometric reconstruction, and rendering quality—and each module targets a specific objective, as a result there are tradeoffs among objectives.
  • Table 4 shows that prototype-based adaptation, supervision gating and semantic uncertainty improve semantic quality. Global keyframes primarily improve reconstruction. As discussed in our responses to R1-Q2 and R2-W3, the orientation constraint likewise trades geometric completeness for semantic consistency. We will clarify these distinct roles and avoid suggesting that every module consistently improves every objective.

W4. Limited Ablation Evaluation.

  • We chose Replica for controlled ablations because its high-fidelity geometry and dense semantic annotations support reliable evaluation of reconstruction and semantic quality. Replica-based ablations are also used in prior work, including SplaTAM (Keetha et al., 2024), ActiveGS (Jin et al., 2025), ActiveSGM (Chen et al., 2025b), and Online Language Splatting (Katragadda et al., 2025).
  • We acknowledge that our four-scene evaluation limits the breadth of the evidence. To strengthen it, we will add ablations on one or two ScanNet++ scenes within the rebuttal period, explicitly state the selection criteria, and report all completed results, although a full eight-scene study is beyond the available time.

Q2. Runtime analysis

Please refer to reseponse to R2(PT1s) W2.

Q3. Concerns about Evaluation Fairness

  • Fig. A3 all trajectories in the same scene start from the same camera pose, we will explicitly mark the starting cameras in the revision.
  • Both methods receive the same maximum frame budget. However, frame count alone does not determine reconstruction quality: active exploration aims to acquire informative views instead of redundant, densely sampled observations. Hence, ActiveLang terminates exploration before exhausting the budgets and therefore uses fewer frames than the passive baselines but achievs better semantic quality.
  • The blue evaluation cameras face outward, rather than toward a shared central point. We will clarify their viewing directions in Fig. A3.
AuthorsOpenReview ↗

Ablation Study on Scannet++

We conduct ablation studies on four ScanNet++ scenes (3f15a9266d, 5eb31827b7, 6115eddb86, and acd95847c5), following the settings in Table 4. Consistent with the Replica ablations, the full pipeline outperforms the ablated variants on eight of the nine evaluation metrics. The best score for each metric is highlighted in bold.

Method2D/SLAM mIoU ↑2D/SLAM Loc ↑2D/NVS mIoU ↑2D/NVS Loc ↑3D mIoU ↑3D mAcc ↑PSNR ↑SSIM ↑LPIPS ↓
w/o Gate26.6749.4221.2739.2817.1123.1522.860.8810.213
w/o Proto27.1850.7322.3241.0420.2928.9723.030.8790.218
w/o Uncert29.1250.1621.4037.9117.6526.2023.290.8820.215
6DoF camera19.7348.7314.5036.1912.9618.1922.390.8720.222
ActiveLang (full)31.4554.0229.2949.6720.4528.2726.120.9280.161
Reviewer_ZBf1OpenReview ↗

Official Review by Reviewer ZBf1

Thank you for the clarifications and additional experiments, including the ScanNet++ ablations. While these address some of my concerns, I remain concerned about the limited methodological novelty, and the earlier reporting inconsistencies still affect my confidence in the experimental evidence. Overall, I will retain my original score.

AuthorsOpenReview ↗

Thanks for the feedback

We thank Reviewer ZBf1 for remaining engaged in this discussion, even if we do not completely agree.

We argue that we are addressing a novel problem, active open-vocabulary semantic mapping, that has been challenging current techniques in various ways. Our methodological novelty centers on two contributions, as detailed in our response to Reviewer PT1s’s W1: (1) learned semantic uncertainty (Eq. 1) which provides an exploration criterion for open-vocabulary setting , where a semantic probability mass function is difficult to specify; (2) prototype-balanced, separation-preserving adaptation (Eqs. 3–4) which addresses the imbalanced semantic stream induced by active exploration, which prior open-vocabulary mappers do not explicitly account for.

We hope that the "earlier reporting inconsistencies" have been fully resolved with our responses and new pdf file. They were due to a copy/paste error in a table caption, which had nothing to do with the data being reported.

Reviewer_PT1sOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ActiveLang targets language-annotated 3D mapping for robot scene understanding. It adopts a compact dual-Gaussian representation for online language-feature adaptation to reconstruct geometry, appearance and open-vocabulary semantics. Guided by semantic uncertainty, its planner chooses informative viewpoints to cut observations and computation.

Critical strengths

(1) This work studies a novel and interesting problem: the combination of active exploration and open-vocabulary mapping. This is the most compelling aspect of the paper.

(2) The experiments demonstrate that the proposed method achieves state-of-the-art performance on several evaluation metrics.

(3) Table 1 is helpful in positioning this work relative to prior methods.

Critical weaknesses

(1) The most critical concern is that this paper reads largely as an application of existing techniques. The online 3DGS representation is adopted from Online Language Splatting; the RGB-D map optimization follows SplaTAM; and the two-stage compression architecture is also inherited from Online Language Splatting. The main claimed contribution, the prototype-based adaptation in Eq. 3, is closely related to prior work in OOD detection and open-set classification (“Distributional Prototype Learning for Out-of-distribution Detection”,”Distributional Prototype Learning for Out-of-distribution Detection”, “Prototype-based Optimal Transport for Out-of-Distribution Detection”), while the separation loss in Eq. 4 is a widely studied approach in metric learning. The paper does not sufficiently articulate what new algorithmic challenges arise in the active exploration setting that cannot be addressed by directly applying these existing techniques.

(2) The mapping system relies on 3DGS, whose computational cost may hinder real-robot deployment. The paper does not report inference time, memory usage, or per-frame processing time, making it difficult to assess practical feasibility.

(3) The 4-DoF constraint, while reasonable under the paper's assumptions, degrades both semantic and rendering quality, as shown in the paper's own ablations (Sec. 4.2 and 4.3). More importantly, it also compromises geometric reconstruction, which is a fundamental component of the map. A further concern is that this constraint limits applicability to real robots, particularly aerial and manipulator platforms that require full 6-DoF motion.

(4) The method requires ground-truth poses, which is not acceptable in real-world environments. This assumption is acknowledged in the conclusion but remains a significant barrier to practical deployment.

Questions for authors

Please refer to weakness.

2 comments
AuthorsOpenReview ↗

Response to R2(PT1s)

Thank you for raising these questions.

W1. Contribution and novelty

We respectfully disagree with this characterization, while acknowledging that the contribution boundaries could be clearer.

  • To the best of our knowledge, ActiveLang is the first system to address active open-vocabulary dense 3D mapping over both objects and background surfaces. Directly combining existing components leaves three gaps. First, existing open-vocabulary mappers, including Online Language Splatting, process observations along externally provided trajectories and do not explicitly address the feature-mode imbalance induced by active exploration: frequently observed surfaces dominate training, while newly encountered content may receive little supervision. Second, their CLIP/SAM-derived semantic supervision cannot be assumed equally reliable across actively selected viewpoints; tilted or inverted observations can produce inconsistent predictions because these models do not explicitly enforce rotation invariance. Third, this inconsistency complicates active selection: closed-set semantic uncertainty criteria do not transfer directly to continuous language features, and uncertainty or mutual-information approximations based on unreliable predictions may fail to identify views that improve semantic reconstruction.
  • Our contributions include prototype-balanced, separation-preserving online adaptation and directly learned semantic uncertainty. The adaptation mechanism addresses streaming imbalance by balancing retained, class-agnostic feature modes and preserving their distinctions during compression; removing it reduces 3D mIoU from 27.34 to 21.12 across four Replica scenes (Table 4). Our learned uncertainty participates in both map learning (Eq.1) and observation acquisition (Eq.5), rather than serving solely as a separately approximated planning signal in most of prior works. We do not claim prototype learning, metric separation, or heteroscedastic modeling individually as novel; our contribution lies in adapting and coupling these mechanisms for active open-vocabulary mapping. We will clarify these distinctions in the revision.

W2. Runtime and memory

  • Inference We provide the following inference runtime and peak GPU memory measurements on Replica. ActiveLang achieves 6.7 FPS for a complete open-vocabulary query, comparable to OnlineLangSplat at 6.8 FPS, with similar peak memory usage.
Inference performance on ReplicaActiveLang (ours)OnlineLangSplatLangSplat
RGB rendering158 FPS540 FPS156 FPS
RGB-D rendering87 FPS233 FPS142 FPS
Language feature rendering52 FPS74 FPS107 FPS
+ Decoding to 768D7.0 FPS7.2 FPS9.8 FPS
Full open-vocabulary query6.7 FPS (150 ms)6.8 FPS9.1 FPS
Peak inference GPU memory6.5 GB6.3 GB7.1 GB
  • Training Appendix A.4.2 reports our hardware configuration and training memory requirements. The system runs on two RTX A6000 GPUs: the language-supervision pipeline uses approximately 6 GB, while the mapping GPU uses approximately 11–25 GB on Replica and peaks at approximately 35 GB depending on scene size. The inference measurements above complement these training-time resource details.

W3. 4‑DoF orientation constraint

  • We clarify that our ablations show that removing the 4-DoF constraint degrades semantic and rendering quality: 2D mIoU drops from 40.91 to 32.33 and PSNR from 33.07 to 30.71 dB (Table 4).
  • As discussed in our response to R1 (jkzQ), Q2, upright observations align with common VLM viewing configurations and familiar object orientations in natural-image training data, helping maintain consistent semantic supervision when language-aligned features lack explicit rotation invariance. We acknowledge that this observation constraint trades geometric completeness for semantic consistency and limits platforms requiring arbitrary camera orientations. Exploring soft orientation constraints that preserve semantic quality while allowing more flexible viewpoints is an important direction for future work.

W4. Ground truth pose assumption

  • Please also read our response to R1 (jkzQ), Q1.
  • Our focus is active open-vocabulary semantic mapping with posed RGB-D inputs, rather than joint pose estimation and mapping. Following the posed-input setting of prior active mapping work, we use ground-truth poses to isolate semantic adaptation and exploration from pose estimation errors. We acknowledge that ground-truth poses are unavailable in practical deployment; integrating an estimated-pose pipeline and evaluating robustness to pose noise and drift remain necessary future steps. We will make this scope and deployment limitation more explicit in the revised manuscript.

Thank you for the author's reply. Most of the concerns have been resolved. This paper provides a good engineering practice, and I look forward to the author open-sourcing the project. I will keep the score.

Reviewer_jkzQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents ActiveLang, a closed‑loop active open‑vocabulary 3D Gaussian mapping system. It uses dual‑Gaussian primitives for geometry‑appearance and semantic feature storage. A prototype‑balanced online autoencoder mitigates feature‑mode imbalance and compression‑induced semantic collapse during sequential active data acquisition. Its planner selects next‑best‑viewpoints combining geometric coverage, travel cost, and predicted semantic uncertainty, with 4‑DoF camera constraint for stable semantic supervision. Evaluations on Replica and ScanNet++ show improved 2D / 3D open‑vocabulary segmentation mIoU compared to offline LangSplat and passive Online Language Splatting baselines, while retaining competitive RGB‑D reconstruction quality. Ablations validate prototype balancing, semantic uncertainty guidance and orientation constraint. Limitations include reliance on ground‑truth camera poses and restricted camera degrees‑of‑freedom.

Critical strengths
  1. Prior open‑vocabulary Gaussian mapping works on pre‑recorded passive trajectories, while existing active mapping mostly targets closed‑vocabulary or pure geometry reconstruction. ActiveLang unifies online semantic feature compression and semantic‑uncertainty‑driven viewpoint selection, filling this research gap, which is highly relevant for embodied‑AI robot scenarios.
  2. The prototype‑balanced online autoencoder explicitly addresses feature‑mode imbalance under uneven actively‑sampled observation streams. Evaluations span synthetic Replica and real‑world ScanNet++, reporting 2D held‑out‑SLAM, zero‑shot novel‑view, full 3D point‑cloud metrics, plus geometric rendering metrics, component ablations and diagnostic analysis of semantic compression failure modes in appendix, not only reporting benchmark numbers but probing underlying failure mechanisms.
Critical weaknesses

The whole pipeline assumes perfect known poses. Real‑world robotic active mapping needs joint SLAM pose estimation; pose drift will corrupt both geometry and open‑vocabulary semantic feature alignment. Without testing under noisy / estimated poses, it remains unclear how the method transfers out of simulation, limiting real‑world practical applicability.

Questions for authors
  1. How would pose estimation noise / drift impact the prototype‑based online autoencoder and semantic uncertainty signals? Have you considered preliminary experiments with noisy poses to understand the degradation boundary?
  2. The 4‑DoF orientation constraint improves semantic performance but limits geometric reconstruction completeness. Are there alternative soft constraints (instead of hard‑fixed roll/pitch) that can preserve semantic quality while allowing more flexible camera orientations?
1 comment
AuthorsOpenReview ↗

Response to R1(jkzQ)

Thank you for raising these questions.

Q1. Pose estimation noise / drift

  • Our focus is active open-vocabulary semantic mapping with posed RGB-D frames, following the input setting adopted by prior active mapping / active semantic mapping methods such as ActiveSplat (Li et al., 2025c), ActiveGAMER (Chen et al., 2025a), AutoNeRF (Marza et al., 2024), ActiveSGM (Chen et al., 2025b), and RayFronts (Alama et al., 2025), rather than active SLAM with joint pose estimation. This controlled setting allows us to study open-vocabulary semantic representation and exploration while avoiding the confounding effects of pose estimation errors on geometric reconstruction and subsequent semantic fusion.
  • Pose noise does not directly affect the prototype-based autoencoder, which operates on image-derived features, but can indirectly alter its observation stream through planning. Current RGBD-SLAM approaches with loop closure detection should be able to overcome the pose noise / drift problem, but it is beyond the scope of this work. We will clarify our research scope in the revised manuscript.

Q2. 4‑DoF orientation constraint

  • Our orientation constraint is motivated by the sensitivity of language-aligned visual features to image orientation. VLM models trained on natural images are exposed predominantly to familiar object orientations and do not explicitly enforce invariance to camera rotation. Strongly tilted or inverted observations can therefore introduce inconsistent semantic supervision, particularly for orientation-dependent categories such as floor and ceiling. Our 4-DoF versus 6-DoF ablation (Table 4 L4 vs. L7) is consistent with this explanation.
  • We agree that a hard orientation constraint trades geometric completeness for semantic consistency. Exploring soft orientation constraints that preserve semantic quality while allowing more flexible viewpoints is an important direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.