acceptodds
Under review as a conference paper at ICLR 2027

A Scene Language Model for Open-Vocabulary Scene Mapping

Abstract

Open-vocabulary 3D scene mapping aims to build a persistent representation of the objects in an environment. Existing systems typically rely on engineered mapping pipelines to associate observations, merge information across views, and maintain a consistent scene representation over time. Many additionally store feature-rich object representations, such as embeddings or image crops, increasing the size and complexity of the persistent memory. We introduce SceneLM, a Scene-Language Model that directly maintains a textual scene map. The full scene is represented as a structured text list of objects, which serves as the model's only persistent memory. For each input image, the model reads the current scene state and updates the map by adding, editing, and removing objects. To learn this behavior, we introduce supervision tasks for iterative scene map maintenance together with an automatic annotation pipeline that generates training data from images without human labels. We evaluate SceneLM on both a language-grounded retrieval benchmark and a localization benchmark. Across both benchmarks, the model produces a scene map that achieves competitive performance with complete mapping systems built from dedicated perception and geometric modules while producing a scene representation that is 6-12x more compact. We further show that SceneLM can be run online on an edge device through experiments on a quadruped. These results show that a persistent open-vocabulary 3D scene map can be maintained directly by a single vision–language model using only a lightweight text representation. Training and inference code is available on the

Primary area:
applications to robotics, autonomy, planning
Keywords:
open-vocabulary 3D scene mapping, vision-language models, object-centric scene representations, language-grounded object retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4vGpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

The paper proposes SceneLM, a vision-language model fine-tuned to maintain a persistent open-vocabulary scene map as a structured text list of objects, locations, and short descriptions. The model predicts ADD, EDIT, and REMOVE operations at each timestep from posed RGB-D observations, replacing explicit geometric backends and feature-rich representations. The paper introduces OVAL, an automatic pseudo-labeling pipeline, and constructs ADD and EDIT-REMOVE supervision tasks. Experiments on open-vocabulary retrieval (ScanNet, HM3D) and navigation localization (Matterport3D) report competitive performance with a 6–12× smaller scene representation compared to feature-rich mapping systems.

Critical strengths
  1. A scalable supervision strategy for scene-map maintenance. OVAL constructs supervision from independent images for adding objects and correcting corrupted map entries, reducing reliance on annotated temporal sequences. Ablations on descriptions, quality filtering, and EDIT/REMOVE training provide evidence that these components contribute to downstream performance. This is a potentially useful practical contribution, although its novelty relative to prior learned-memory approaches remains insufficiently established.

  2. Evidence that compact text-only scene memory can support useful downstream tasks. The paper evaluates retrieval and localization across ScanNet, HM3D, and Matterport3D, and demonstrates online deployment on a robot. It reports approximately 6–12× lower persistent scene-memory storage than FARM while retaining competitive task performance. These results substantiate the practical utility of the representation, although the advantage depends on the evaluation protocol and does not extend to mapping latency.

Critical weaknesses
  1. The substantive novelty relative to the closest prior work is not established. EPM (Planning with an Embodied Learnable Memory, ICLR 2026) already trains a single VLM to maintain textual entity records containing descriptions and 3D locations through Add/Update/Remove operations. Although EPM targets planning and differs in its action conditioning and initialization, evaluating a related update framework on mapping benchmarks does not, by itself, establish methodological novelty. The submission neither cites nor discusses EPM, and does not clearly identify the substantive advance provided by independent-image supervision and corruption-based training. This leaves its central contribution insufficiently distinguished from existing learned textual-memory approaches.

  2. The experiments do not sufficiently isolate the benefit of learned map updates. Comparisons with FARM simultaneously change the representation, perception pipeline, and update mechanism. TagMap provides a relevant precedent for text-based mapping without a learned update policy, but the existing localization comparison in Table 3 does not control these differences. A baseline using comparable textual object records and perception quality, but rule-based association and updates, would help isolate the contribution. The ADD-only ablation supports EDIT/REMOVE training, while the Fix-tasks metrics in Appendix D.2 evaluate corrections to synthetically corrupted records; neither directly establishes consistency during continuous mapping. Quantitative evaluations of duplicate insertion, incorrect merging, erroneous deletion, and association upon revisiting would therefore be important for substantiating the claimed map-maintenance capability.

Questions for authors

See weakness.

4 comments
AuthorsOpenReview ↗

4vGp [1/3]

Thank you for your time and effort in reviewing our paper. We appreciate your recognition of the scalable supervision strategy and the practical utility of compact textual scene maps. We generally agree with your concerns and think that the following clarifications/modifications improve the manuscript.

CW1 - Similarities with EPM (ICLR-2026)

Thank you for bringing Planning with an Embodied Learnable Memory (EPM, ICLR 2026) to our attention. We were unaware of this work when preparing the submission and agree that it should have been cited and discussed. EPM and SceneLM share the formulation of a VLM maintaining textual entity records through Add/Update/Remove operations. We will update the manuscript to acknowledge this shared formulation explicitly and revise the contribution statement to emphasize our scalable supervision approach, training-tasks setup, and the resulting real-world mapping capabilities.

As you highlight among the strengths, SceneLM's supervision strategy enables training from independent images. We believe this is the central methodological contribution relative to EPM: the combination of OVAL's automatic annotation pipeline and the construction of map-maintenance training examples removes the need for simulator trajectories, privileged temporal object identities, or manually annotated 3D scene maps. EPM itself identifies real-world training as a future direction (§6), motivated by the difficulty of collecting annotated data.

We also believe there is an important difference in initialization. Our understanding of EPM is that its memory begins with supplied furniture information. The supplement's general model description (§A.2.1, p. 15) states:

“The Entity List is initialized with the furniture in the environment, with the 3D coordinate corresponding to the furniture centroid and the description corresponding to the furniture name.”

The real-world experiment description (§A.10) also specifies a hand-annotated house layout supplied to EPM and its planner. SceneLM begins with an empty semantic object list and constructs its contents from incoming posed RGB-D observations.

We additionally train the model to produce short appearance descriptions alongside category labels, and our description ablation supports their usefulness for downstream retrieval. EPM's documented training targets use category names supplemented with relationship and state information (§A.2.3). Our appearance descriptions aim to provide a complementary source of information about individual objects, such as color, material, and shape. We also show that this information yields improvements in the evaluated downstream tasks.

Our experiments establish the practical value of this approach for large-scale open-vocabulary mapping, and the robot experiments demonstrate onboard deployment with reasonable latency. Through the release of model checkpoints, training recipes, and the data-generation pipeline, we also aim to make this family of methods easier to reproduce, use, and extend. Together, despite its conceptual similarities with EPM, we believe SceneLM makes a significant contribution, and provides further insights, to advancing this research field.

AuthorsOpenReview ↗

4vGp [2/3]

CW 2 - Isolate benefit of map updates

As you note, the ADD-only ablation supports the benefit of EDIT/REMOVE training for SceneLM, and the localization ablations in supplementary Table 14 show the same trend as the retrieval ablations in Table 2. We agree that this does not compare the learned update policy with a rule-based association and update. Following your suggestion, we additionally compare SceneLM with an engineered system that uses the same model for independent per-frame object detection, while delegating association and map updates to a separate rule-based method in CW2.1. We also try to further quantify the map-maintenance capabilities in CW2.2

CW 2.1 - Comparison with rule-based object update

We use the same SceneLM-2B checkpoint as an independent object detector on each frame, producing the same type of textual object records, and maintain the map through rule-based/geometric fusion. Detections with matching category labels are associated one-to-one (using a linear assignment solver maximizing the number of valid matches and then minimizes their total 3D distance) within a spherical radius r. Matched positions are averaged using observation counts, and the latest description is retained. Unmatched detections create new entries, and absence alone does not delete an entry. (Objects can therefore be merged, but not removed) Both methods use the same input frames, camera poses, depth, and downstream retrieval protocol. This controls the model weights and representation, although independent detection ("Indep.") also removes the existing-map context from ADD (in a sense, SceneLM always believe the map is empty), so the comparison does not isolate EDIT/REMOVE alone.

DatasetMethodRadius (m)Entries/sceneAcc@1 (%)R@5 (%)R@10 (%)
ScanNetIndep. ADD + fusion0.25363.137.7973.8183.15
ScanNetIndep. ADD + fusion0.50280.536.2376.6584.12
ScanNetIndep. ADD + fusion0.75250.437.0377.1783.45
ScanNetIndep. ADD + fusion1.00233.036.7876.1882.34
ScanNetIndep. ADD + fusion1.50209.735.3271.9378.55
ScanNetIndep. ADD + fusion2.00192.231.6465.8871.60
ScanNetIndep. ADD + fusion2.50179.529.0661.4366.94
ScanNetSceneLM—236.838.6770.2078.28
HM3DIndep. ADD + fusion0.252,079.86.9318.8127.50
HM3DIndep. ADD + fusion0.501,573.16.7419.2428.63
HM3DIndep. ADD + fusion0.751,356.16.6419.3529.12
HM3DIndep. ADD + fusion1.001,229.86.5619.2029.29
HM3DIndep. ADD + fusion1.501,070.36.2519.1829.22
HM3DIndep. ADD + fusion2.00963.86.0918.7627.86
HM3DIndep. ADD + fusion2.50884.05.9118.5127.16
HM3DSceneLM—817.86.7218.2526.47

The comparison shows a trade-off depending on the chosen radius. On ScanNet, SceneLM has higher Acc@1 at every tested radius, while the baseline can have higher recall if r is set properly. At similar map sizes (236.8 versus 233.0 entries at r = 1 m), SceneLM has higher Acc@1 (38.67% versus 36.78%), while fusion has higher R@10 (82.34% versus 78.28%). On HM3D, fusion at r = 0.5 m approximately matches SceneLM's Acc@1 (6.74% versus 6.72%) using 1,573.1 versus 817.8 entries, and has higher R@10 (28.63% versus 26.47%). Fusion is therefore competitive, particularly for recall. However, independent per-frame detection may retain additional object hypotheses that improve recall, since its proposals are not conditioned on the existing map. The above comparison does not isolate that effect from association and updating.

These results show a trade-off between map size and retrieval performance for different fusion parameters. SceneLM offers a useful combination of map size and retrieval accuracy without a separate engineered and manually tuned tracking/fusion method. Reducing reliance on such hand-specified association rules is a motivation for our approach, and we believe focusing more on learned fusion/tracking is an interesting research direction moving forward.

AuthorsOpenReview ↗

4vGp [3/3]

CW 2.2 Cross-view association

To directly examine cross-view association, we additionally evaluated whether an already represented object receives a duplicate entry when observed from another viewpoint (0.6-3.3s away), following the SAM3-based evaluation suggested by reviewer GbzQ. Each image pair starts with a fresh map constructed from the first image. SceneLM then processes the second image using its map context and ADD/EDIT/REMOVE operations. SAM3 tracks supply reference identities for evaluation and is never provided to SceneLM. We evaluated 250 image pairs from each dataset. Scoring requires a usable SAM3 reference, valid model responses, and an existing object record that is included in the second image's context and projects inside the tracked mask.

DatasetInitial mapped observationsEligible observationsPreserved without duplicationDuplicate insertionLost original associationsFaulty removalsEdits to outside mask
ScanNet1,867704652 (92.61%)52 (7.39%)0 (0%)0 (0%)0 (0%)
HM3D1,455443418 (94.36%)25 (5.64%)0 (0%)0 (0%)0 (0%)

Among these eligible observations, SceneLM preserves the existing object without introducing a duplicate in 92.6% of ScanNet cases and 94.4% of HM3D cases. No eligible original entry is removed or has its point moved outside the tracked mask. This provides further evidence of cross-view association on image pairs despite training from independent images, while also quantifying remaining duplicate insertions.

Reviewer_4vGpOpenReview ↗

Final comments

Thank you for the thorough response and the additional experiments, and for reporting them candidly even where they favor the baseline. My soundness concerns are largely addressed and no longer factor into my rating.

My remaining reservation is about significance. EPM (ICLR 2026) is prior work sharing the core formulation, which narrows the contribution to the supervision pipeline — real and useful, but incremental. The new fusion comparison also suggests that, with perception controlled, the benefit of learned updates is mainly a more compact map without manual tuning rather than better mapping performance.

If a borderline score were available, I would lean toward it, as this is a close call and the work is honest and practically useful. Since the scale does not offer one, I settle on a weak reject (2), with the reservation being novelty and significance rather than correctness.

Reviewer_r6e9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SceneLM, a scene-language model that represents a scene using a textual representation. Unlike conventional open-vocabulary mapping systems, the proposed method uses a single VLM to maintain and update a compact textual scene map given posed RGB-D observations and the existing scene map. To this end, the model is trained on map-maintenance tasks that predict simple operations such as ADD, EDIT, and REMOVE. As new observations arrive, it autoregressively generates commands to update the map incrementally. Experimental results show competitive performance against existing methods on language-grounded object retrieval and 3D object localization while using less scene memory.

Critical strengths
  • Problem formulation: The authors describe the map as a compact textual representation consisting of 3D object positions, labels, and short descriptions, and formulate persistent scene-map maintenance as predicting ADD, EDIT, and REMOVE operations over this map. This strategy provides a unified formulation for handling object insertion, cross-view object association, and map correction with a single VLM. This design reduces reliance on separate association and map-maintenance modules and eliminates the need to persistently store visual embeddings or image crops.

  • Automatic training data construction: They introduce OVAL to automatically generate 2D object locations, labels, and descriptions from individual images. These annotations are then used to construct map-maintenance supervision. This approach enables training on ADD, EDIT, and REMOVE operations without manually annotated 3D scene maps and allows the training data to be scaled using images from diverse datasets.

Critical weaknesses
  • Limitations of the map representation: The proposed textual state is memory-efficient and effective for the evaluated retrieval and localization tasks. However, it may not explicitly model relationships between objects, leaving it unclear whether it is sufficient for complex queries or downstream reasoning where relational information is important. Scene graphs, which explicitly represent inter-object relationships, may be more suitable for such tasks. From this perspective, relation-centric evaluations might be needed to establish the scope and limitations of the proposed representation.

  • Sensitivity to pose and depth errors: The proposed method uses posed RGB-D observations to compute the 3D world positions of objects and select existing map entries to present to the VLM for the current observation. Thus, pose and depth errors may affect not only object localization accuracy but also cross-view association and ADD, EDIT, and REMOVE decisions. Although the authors associate the increased number of duplicate objects in real-world experiments with pose uncertainty, they do not systematically evaluate how mapping performance changes with different levels of error. This makes it difficult to assess the robustness of the proposed framework.

Questions for authors

If any of the questions below are already addressed in the paper or supplementary material, I would appreciate pointers to the relevant sections.

  • When an EDIT operation produces a new 3D position, does it replace the existing position, or are the two estimates combined? If they are combined, what update rule is used?
  • Although the supplementary material discusses duplicate entries and shows examples of their removal, multiple cubes in the right-hand 3D visualization of the supplementary video "build_map_with_retrieval_scene_0000_00.mp4" appear to correspond to the same physical object, such as a chair or rug. I understand that each small cube represents an individual object entry; if this interpretation is correct, a considerable number of duplicate entries seem to remain. Given that the proposed method is trained to remove duplicates through the VLM’s REMOVE operation, could the authors explain why these duplicates persist? Is pose uncertainty, as discussed in the paper, the primary reason for these remaining duplicates? Could the authors also clarify whether these are temporary duplicates that are removed with subsequent observations or duplicates that remain in the final map?
2 comments
AuthorsOpenReview ↗

r6e9 [1/1]

Thank you for the constructive feedback. We address the representation's scope, sensitivity to geometry errors, position-update rule, and persistent duplicates below.

CW1 - Explicit Relations in map

Our evaluation includes relational object grounding. ReferIt3D’s SR3D expressions distinguish targets through spatial relationships to anchor objects, e.g. “The chair under the chalkboard” (Achlioptas et al., §3.1 and Fig. 1). IRef-VLA similarly specifies that every generated statement contains at least one relation (Zhang et al., §IV-D).

We agree that SceneLM does not store explicit inter-object relation edges. However, some spatial relationships can be computed from the stored object positions when answering a query. The retrieval method from FARM already uses this information, for example, to compare distances between candidate objects and an anchor object. These relationships therefore do not need to be stored explicitly during mapping.

The short descriptions can also contain relational information. For example, generated maps include “White paper with a black pen resting diagonally across it” and “Black cylindrical trash can with white plastic bag inside.” These examples show that the representation can retain such information, although we do not explicitly supervise its consistent inclusion. One possible extension is to adapt OVAL’s object describer, DescribeAnything, to emphasize inter-object relationships and evaluate whether this improves downstream performance. Nevertheless, the stored points and descriptions may omit information needed for more complex queries. Whether explicitly storing inter-object relations improves performance over our point-and-text representation remains an open question for future work.

CW 2 - Pose/Depth sensitivity

Thank you for suggesting this experiment. We conduct a controlled sensitivity study using the main SceneLM-2B on five scenes per dataset, with two noise seeds. We rebuild the maps with perturbed camera poses and depth, affecting both map-context selection and 3D position updates, and evaluate retrieval against unchanged ground truth. We test four synthetic noise levels, specified by the root mean square (RMS) translation and rotation perturbations and the standard deviation of multiplicative Gaussian depth noise.

Combined conditionTranslation RMSERotation RMSERelative depth-noise standard deviation
Clean0 cm0°0%
L11 cm0.5°2%
L23 cm1°3%
L35 cm2°5%
L410 cm5°10%

The following table reports retrieval performance and the percentage change in the number of map entries relative to the clean condition:

ConditionScanNet Acc@1ScanNet R@10ScanNet map growthHM3D Acc@1HM3D R@10HM3D map growth
Clean31.7%75.8%0.0%5.9%24.8%0.0%
L131.3%73.4%+0.5%4.9%22.0%+4.2%
L231.0%73.2%+5.5%4.6%21.2%+14.6%
L326.5%71.5%+23.5%3.4%19.2%+32.6%
L417.4%59.5%+81.9%3.0%13.6%+90.8%

These results show the sensitivity to geometry errors, where more noise leads to lower performance. We note that the number of objects increases with the noise, and while not a measure of duplicate rate, it could be seen as an indication that the amount of duplicates is correlated with the noise level.

Q1 - Updating 3D position

When editing an object's position, we replace the old position with the new position. We see no issue with using another update rule, but we motivate our decision with that we want as few engineered components as possible.

Q2 - Duplicate objects

Some duplicate entries are removed during mapping, but others remain in the final map. We believe an important contributor is partial visibility, where an object can be detected again while its previously mapped location lies outside the current view. Pose/depth errors and association mistakes can also contribute. We quantified repeated entries across all validation scenes by matching predictions to GT bounding boxes using their positions and exact labels. Under this diagnostic, 11.0% of GT boxes in ScanNet and 1.35% in HM3D contain two or more matching predictions. These percentages measure repeated representation under the matching rule, rather than verified physical duplication. The supplementary video shows a ScanNet scene, which suffers more from duplicates than HM3D. We hypothesize that the typically closer camera viewpoints, which produce more partial views of objects, contribute to the greater repeated representation observed in ScanNet. We provide two other types of analysis of duplicates in our response to GbzQ (Q1 & Q2).

Thank you for the response. The rebuttal has addressed my concerns regarding the explicit relations between objects in the generated maps and sensitivity to pose and depth errors. Also, the authors’ answers to my questions are reasonable. Thus, I maintain my original rating, Weak acceptance.

Reviewer_GbzQOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SceneLM, a fine-tuned VLM that maintains a persistent open-vocabulary 3D scene map as a structured text list of objects, each with a category label, a short description and a 3D position. The model processes posed RGB-D observations as a stream, with the text map as its only persistent memory. It takes the incoming frame and the currently visible part of the map to predicts the ADD, EDIT, and REMOVE operations to update the map. To train the model without 3D annotations, the authors build an automatic pipeline that produces a large number of object pseudo-labels on numerous images, and algorithmically construct supervision for different tasks. The evaulation covers single-frame map updates against off-the-shelf VLMs, language-grounded object retrieval and localization downstream tasks, and report efficiency.

Critical strengths
  1. Representing the persistent 3D scene map as a list of text and turning the map update into a operation predicted by a VLM is novel to me. Furthermore, the supervision design without the need of 3D annotation looks solid and cover a wide range and large number of images and objects, calibrated the judge of the automatic-annotated dataset with human labels increase the soundness.

  2. Clear presentation. The paper is easy to follow and read in a single pass. The problem, representation and the expreiment designs are introduced with enough overview so the audience rarely has to go back and forth. Training tasks, metrics and evaluation protocols are presented in details, and the appendix further enrich the implementation details.

  3. The on-robot deployment showcase is interesting. And the report efficiency is transparent. The video in supplementary looks good and further improve my understanding of this paper.

Critical weaknesses
  1. [Resolved, honestly discissued] Duplicate entries are accumulating in the map. The current experiment did not include metrics or in-depth analysis for this. The supplementary video on ScanNet scene0000_00 demonstrates that duplicate entries are continuously added to the map, while EDIT and REMOVE operations are relatively infrequent. At the end,the model reports 304 objects in the scene, while from ScanNet instance map ground truth, there should be roughly 56 objects without stuff. For instance, multiple guitar entries in the map, while there is only one guitar. This duplicate issue are presented in teh main text as something that the model can corrects, but acknowledges as a failure mode in the appendix. I think this issue effects the correctness of the map, and should be quantified in the main paper. The current metrics, however, seems not enough to measure such effect.

  2. [Affected the article's claim] SceneLM is trained from OVAL's pseudo-label, yet not compared with the straightforward baseline on OVAL itself. Its VLM baselines compared against are off-the-shelf, lack the knowledge from this model. It would be interesting to see comparison that use geometry and pose to fuse OVAL detection into a map, and measure the difference.

Questions for authors

Q1 (W1): Can you report the map level metrics against GT instance information provided by the dataset to compare SceneLM against the baselines? I am particularly interested in the duplicate rate.

Q2 (W1): Training are done on the single image. Can you evaluate association on real-world cross-view pairs where you have consistent object identities (from GT or SAM track) to see if SceneLM correctly keep or edit the objects?

Q3 (W2): It will be nice to see a comparison against the OVAL baseline.

3 comments
AuthorsOpenReview ↗

GbzQ [1/2]

Thank you for the constructive review.

You are right that duplicate entries persist in the final maps and that our existing retrieval metrics do not directly quantify this limitation. Although SceneLM is trained to remove duplicates, it does not always do so successfully. We will make this limitation explicit in the main text.

The repeated entries for the same guitar in scene0000_00 are a clear example of this failure. However, the difference between the 304 map entries and the roughly 56 annotated instances does not directly measure duplication since SceneLM's open-vocabulary output can include objects without separate instance annotations in ScanNet.

Below, we report additional map-level diagnostics (Q1) and a cross-view association evaluation (Q2) to better characterize this behavior. We also compare against OVAL mapping baselines (Q3).

Q1 - GT-based map diagnostic

We here evaluate object duplication and coverage in the final SceneLM-2B maps across 30 ScanNet scans and 30 HM3D scenes using the GT boxes.

To estimate duplicate entries, we associate each map entry with a GT bounding box if its predicted point lies inside a same (casefolding and collapsing whitespace) noun phrase. A box receiving k assigned predictions contributes max(k − 1, 0) duplicate entries. Their sum is divided by all final map entries.

We also measure coverage: the fraction of GT boxes containing at least one predicted point, using three increasingly strict label criteria:

  • Any label: no label restriction.
  • Training-pool match: the same normalized noun phrase, or either label directly lists the other in the existing training synonym pool. This includes synonyms and broader/narrower terms.
  • Same noun: the entire noun phrase matches after casefolding and collapsing whitespace, without aliases or head-noun matching.
DatasetEntries/sceneEstimated duplicate entries (%)Coverage: any labelCoverage: training-pool matchCoverage: same noun
ScanNet236.8792 / 7,105 = 11.15%1,416 (77.46%)955 (52.24%)607 (33.21%)
HM3D817.81,393 / 24,535 = 5.68%10,591 (46.45%)5,493 (24.09%)2,716 (11.91%)

The duplicate entries fraction is higher on ScanNet than on HM3D (11.15% versus 5.68%), and the ScanNet visualizations in sup.mat. illustrate persistent duplicates. We suspect that partial views of large objects contribute the most to this failure: an object's stored 3D point can project outside the current camera field of view even while part of the object remains visible. Its record is then absent from the model's context, allowing a new entry to be added. Pose and depth errors, as well as association mistakes by the model, also contribute...

One possible mitigation is to include nearby out-of-view object records in the model’s context, allowing association with previously mapped objects even when only part of the object is visible. However, despite these duplicates, the maps created by SceneLM remain useful for the evaluated grounding tasks: a retrieved entry is considered correct if its predicted point lies inside the target object’s bounding box, even when other entries represent the same object.

Q2 - Cross-view association

Thank you for this suggestion. We agree that evaluating association across viewpoints provides a useful complement to Table 1. We therefore evaluate whether an object already represented in the map receives a duplicate entry when observed from another viewpoint, randomly sampled 0.6–3.3 seconds later.

Each image pair starts with a fresh map constructed by (empty context) SceneLM from the first image (image A). SceneLM then processes the second image (image B) using the map context from Image A and ADD/EDIT/REMOVE operations. SAM3 tracks objects through all intermediate frames between A and B and supplies reference identities for evaluation. SceneLM sees only images A and B and receives no SAM3 tracking information. We evaluated 250 randomly sampled image pairs from each dataset. An object observation is eligible for scoring when the SAM3 reference and model responses are valid, and the existing map entry is included in image B’s context with its projected point inside the tracked object mask.

DatasetInitial mapped observationsEligible observationsPreserved without duplicationDuplicate insertionLost original associationsFaulty removalsEdits to outside mask
ScanNet1,867704652 (92.61%)52 (7.39%)0 (0%)0 (0%)0 (0%)
HM3D1,455443418 (94.36%)25 (5.64%)0 (0%)0 (0%)0 (0%)

SceneLM introduces duplicate entries in 7.4% of eligible object observations on ScanNet and 5.6% on HM3D. In the remaining cases, the existing object is retained without adding a duplicate. No eligible original entry is removed or has its point moved outside the tracked mask.

AuthorsOpenReview ↗

GbzQ [2/2]

Q3 - OVAL as baseline

We agree that OVAL is an important baseline. We generated quality-checked OVAL detections on all evaluation frames, lifted their points using the same depth and poses as SceneLM, and built two map variants: accumulation of every accepted observation, and rule-based fusion.

The fusion baseline matches detections to existing map entries with the same normalized label within a predefined radius. Matching is one-to-one, using a linear assignment solver that maximizes the number of valid matches and then minimizes their total 3D distance. Each matched entry retains its identity, updates its position to the mean of its associated observations, and receives the latest description. Unmatched detections create new entries.

We report the fixed 0.5 m reference setting here and will include the complete 0.25–2.5 m sensitivity sweep in sup. mat. No radius is uniformly best across metrics.

DatasetMethodMean entries/sceneAcc@1R@5R@10Inference time (ms/frame)
ScanNetOVAL accumulation1,11642.570.880.18,650
ScanNetOVAL + fusion, 0.5 m35940.878.785.28,650
ScanNetSceneLM23738.770.278.3965
HM3DOVAL accumulation4,8897.518.826.27,940
HM3DOVAL + fusion, 0.5 m1,9426.620.530.17,940
HM3DSceneLM8186.718.326.5843

OVAL with fusion outperforms SceneLM on all three ScanNet retrieval metrics and both HM3D recall metrics, while HM3D Acc@1 is similar. OVAL-maps have approximately 1.5x as many map entries on ScanNet and 2.4x on HM3D. SceneLM therefore produces smaller maps, while OVAL fusion achieves higher retrieval performance on most metrics. Importantly, OVAL requires significantly more VRAM and is about 9x slower per frame.

Thank you for the honest analysis of the duplicate behavior, including the cause analysis and the cross-view association study. These results do not affect the key strengths that led to my positive assessment: the text-based map representation with VLM-predicted update operations, the annotation-free supervision pipeline.

The OVAL comparison clarifies that SceneLM’s contribution lies in map compactness and a single, online model rather than in retrieval accuracy, where rule-based fusion of OVAL detections performs better.

Having gained a clearer understanding of both the strengths and limitations of the method, I still find this paper informaive and insightful, so I keep my rating at 3 and raise my confidence to accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.