- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
The paper proposes SceneLM, a vision-language model fine-tuned to maintain a persistent open-vocabulary scene map as a structured text list of objects, locations, and short descriptions. The model predicts ADD, EDIT, and REMOVE operations at each timestep from posed RGB-D observations, replacing explicit geometric backends and feature-rich representations. The paper introduces OVAL, an automatic pseudo-labeling pipeline, and constructs ADD and EDIT-REMOVE supervision tasks. Experiments on open-vocabulary retrieval (ScanNet, HM3D) and navigation localization (Matterport3D) report competitive performance with a 6–12× smaller scene representation compared to feature-rich mapping systems.
Critical strengths
-
A scalable supervision strategy for scene-map maintenance. OVAL constructs supervision from independent images for adding objects and correcting corrupted map entries, reducing reliance on annotated temporal sequences. Ablations on descriptions, quality filtering, and EDIT/REMOVE training provide evidence that these components contribute to downstream performance. This is a potentially useful practical contribution, although its novelty relative to prior learned-memory approaches remains insufficiently established.
-
Evidence that compact text-only scene memory can support useful downstream tasks. The paper evaluates retrieval and localization across ScanNet, HM3D, and Matterport3D, and demonstrates online deployment on a robot. It reports approximately 6–12× lower persistent scene-memory storage than FARM while retaining competitive task performance. These results substantiate the practical utility of the representation, although the advantage depends on the evaluation protocol and does not extend to mapping latency.
Critical weaknesses
-
The substantive novelty relative to the closest prior work is not established. EPM (Planning with an Embodied Learnable Memory, ICLR 2026) already trains a single VLM to maintain textual entity records containing descriptions and 3D locations through Add/Update/Remove operations. Although EPM targets planning and differs in its action conditioning and initialization, evaluating a related update framework on mapping benchmarks does not, by itself, establish methodological novelty. The submission neither cites nor discusses EPM, and does not clearly identify the substantive advance provided by independent-image supervision and corruption-based training. This leaves its central contribution insufficiently distinguished from existing learned textual-memory approaches.
-
The experiments do not sufficiently isolate the benefit of learned map updates. Comparisons with FARM simultaneously change the representation, perception pipeline, and update mechanism. TagMap provides a relevant precedent for text-based mapping without a learned update policy, but the existing localization comparison in Table 3 does not control these differences. A baseline using comparable textual object records and perception quality, but rule-based association and updates, would help isolate the contribution. The ADD-only ablation supports EDIT/REMOVE training, while the Fix-tasks metrics in Appendix D.2 evaluate corrections to synthetically corrupted records; neither directly establishes consistency during continuous mapping. Quantitative evaluations of duplicate insertion, incorrect merging, erroneous deletion, and association upon revisiting would therefore be important for substantiating the claimed map-maintenance capability.
Questions for authors
See weakness.
4 comments
4vGp [2/3]
CW 2 - Isolate benefit of map updates
As you note, the ADD-only ablation supports the benefit of EDIT/REMOVE training for SceneLM, and the localization ablations in supplementary Table 14 show the same trend as the retrieval ablations in Table 2. We agree that this does not compare the learned update policy with a rule-based association and update. Following your suggestion, we additionally compare SceneLM with an engineered system that uses the same model for independent per-frame object detection, while delegating association and map updates to a separate rule-based method in CW2.1. We also try to further quantify the map-maintenance capabilities in CW2.2
CW 2.1 - Comparison with rule-based object update
We use the same SceneLM-2B checkpoint as an independent object detector on each frame, producing the same type of textual object records, and maintain the map through rule-based/geometric fusion. Detections with matching category labels are associated one-to-one (using a linear assignment solver maximizing the number of valid matches and then minimizes their total 3D distance) within a spherical radius r. Matched positions are averaged using observation counts, and the latest description is retained. Unmatched detections create new entries, and absence alone does not delete an entry. (Objects can therefore be merged, but not removed) Both methods use the same input frames, camera poses, depth, and downstream retrieval protocol. This controls the model weights and representation, although independent detection ("Indep.") also removes the existing-map context from ADD (in a sense, SceneLM always believe the map is empty), so the comparison does not isolate EDIT/REMOVE alone.
| Dataset | Method | Radius (m) | Entries/scene | Acc@1 (%) | R@5 (%) | R@10 (%) |
|---|---|---|---|---|---|---|
| ScanNet | Indep. ADD + fusion | 0.25 | 363.1 | 37.79 | 73.81 | 83.15 |
| ScanNet | Indep. ADD + fusion | 0.50 | 280.5 | 36.23 | 76.65 | 84.12 |
| ScanNet | Indep. ADD + fusion | 0.75 | 250.4 | 37.03 | 77.17 | 83.45 |
| ScanNet | Indep. ADD + fusion | 1.00 | 233.0 | 36.78 | 76.18 | 82.34 |
| ScanNet | Indep. ADD + fusion | 1.50 | 209.7 | 35.32 | 71.93 | 78.55 |
| ScanNet | Indep. ADD + fusion | 2.00 | 192.2 | 31.64 | 65.88 | 71.60 |
| ScanNet | Indep. ADD + fusion | 2.50 | 179.5 | 29.06 | 61.43 | 66.94 |
| ScanNet | SceneLM | — | 236.8 | 38.67 | 70.20 | 78.28 |
| HM3D | Indep. ADD + fusion | 0.25 | 2,079.8 | 6.93 | 18.81 | 27.50 |
| HM3D | Indep. ADD + fusion | 0.50 | 1,573.1 | 6.74 | 19.24 | 28.63 |
| HM3D | Indep. ADD + fusion | 0.75 | 1,356.1 | 6.64 | 19.35 | 29.12 |
| HM3D | Indep. ADD + fusion | 1.00 | 1,229.8 | 6.56 | 19.20 | 29.29 |
| HM3D | Indep. ADD + fusion | 1.50 | 1,070.3 | 6.25 | 19.18 | 29.22 |
| HM3D | Indep. ADD + fusion | 2.00 | 963.8 | 6.09 | 18.76 | 27.86 |
| HM3D | Indep. ADD + fusion | 2.50 | 884.0 | 5.91 | 18.51 | 27.16 |
| HM3D | SceneLM | — | 817.8 | 6.72 | 18.25 | 26.47 |
The comparison shows a trade-off depending on the chosen radius. On ScanNet, SceneLM has higher Acc@1 at every tested radius, while the baseline can have higher recall if r is set properly. At similar map sizes (236.8 versus 233.0 entries at r = 1 m), SceneLM has higher Acc@1 (38.67% versus 36.78%), while fusion has higher R@10 (82.34% versus 78.28%). On HM3D, fusion at r = 0.5 m approximately matches SceneLM's Acc@1 (6.74% versus 6.72%) using 1,573.1 versus 817.8 entries, and has higher R@10 (28.63% versus 26.47%). Fusion is therefore competitive, particularly for recall. However, independent per-frame detection may retain additional object hypotheses that improve recall, since its proposals are not conditioned on the existing map. The above comparison does not isolate that effect from association and updating.
These results show a trade-off between map size and retrieval performance for different fusion parameters. SceneLM offers a useful combination of map size and retrieval accuracy without a separate engineered and manually tuned tracking/fusion method. Reducing reliance on such hand-specified association rules is a motivation for our approach, and we believe focusing more on learned fusion/tracking is an interesting research direction moving forward.
4vGp [3/3]
CW 2.2 Cross-view association
To directly examine cross-view association, we additionally evaluated whether an already represented object receives a duplicate entry when observed from another viewpoint (0.6-3.3s away), following the SAM3-based evaluation suggested by reviewer GbzQ. Each image pair starts with a fresh map constructed from the first image. SceneLM then processes the second image using its map context and ADD/EDIT/REMOVE operations. SAM3 tracks supply reference identities for evaluation and is never provided to SceneLM. We evaluated 250 image pairs from each dataset. Scoring requires a usable SAM3 reference, valid model responses, and an existing object record that is included in the second image's context and projects inside the tracked mask.
| Dataset | Initial mapped observations | Eligible observations | Preserved without duplication | Duplicate insertion | Lost original associations | Faulty removals | Edits to outside mask |
|---|---|---|---|---|---|---|---|
| ScanNet | 1,867 | 704 | 652 (92.61%) | 52 (7.39%) | 0 (0%) | 0 (0%) | 0 (0%) |
| HM3D | 1,455 | 443 | 418 (94.36%) | 25 (5.64%) | 0 (0%) | 0 (0%) | 0 (0%) |
Among these eligible observations, SceneLM preserves the existing object without introducing a duplicate in 92.6% of ScanNet cases and 94.4% of HM3D cases. No eligible original entry is removed or has its point moved outside the tracked mask. This provides further evidence of cross-view association on image pairs despite training from independent images, while also quantifying remaining duplicate insertions.
Final comments
Thank you for the thorough response and the additional experiments, and for reporting them candidly even where they favor the baseline. My soundness concerns are largely addressed and no longer factor into my rating.
My remaining reservation is about significance. EPM (ICLR 2026) is prior work sharing the core formulation, which narrows the contribution to the supervision pipeline — real and useful, but incremental. The new fusion comparison also suggests that, with perception controlled, the benefit of learned updates is mainly a more compact map without manual tuning rather than better mapping performance.
If a borderline score were available, I would lean toward it, as this is a close call and the work is honest and practically useful. Since the scale does not offer one, I settle on a weak reject (2), with the reservation being novelty and significance rather than correctness.
4vGp [1/3]
Thank you for your time and effort in reviewing our paper. We appreciate your recognition of the scalable supervision strategy and the practical utility of compact textual scene maps. We generally agree with your concerns and think that the following clarifications/modifications improve the manuscript.
CW1 - Similarities with EPM (ICLR-2026)
Thank you for bringing Planning with an Embodied Learnable Memory (EPM, ICLR 2026) to our attention. We were unaware of this work when preparing the submission and agree that it should have been cited and discussed. EPM and SceneLM share the formulation of a VLM maintaining textual entity records through Add/Update/Remove operations. We will update the manuscript to acknowledge this shared formulation explicitly and revise the contribution statement to emphasize our scalable supervision approach, training-tasks setup, and the resulting real-world mapping capabilities.
As you highlight among the strengths, SceneLM's supervision strategy enables training from independent images. We believe this is the central methodological contribution relative to EPM: the combination of OVAL's automatic annotation pipeline and the construction of map-maintenance training examples removes the need for simulator trajectories, privileged temporal object identities, or manually annotated 3D scene maps. EPM itself identifies real-world training as a future direction (§6), motivated by the difficulty of collecting annotated data.
We also believe there is an important difference in initialization. Our understanding of EPM is that its memory begins with supplied furniture information. The supplement's general model description (§A.2.1, p. 15) states:
The real-world experiment description (§A.10) also specifies a hand-annotated house layout supplied to EPM and its planner. SceneLM begins with an empty semantic object list and constructs its contents from incoming posed RGB-D observations.
We additionally train the model to produce short appearance descriptions alongside category labels, and our description ablation supports their usefulness for downstream retrieval. EPM's documented training targets use category names supplemented with relationship and state information (§A.2.3). Our appearance descriptions aim to provide a complementary source of information about individual objects, such as color, material, and shape. We also show that this information yields improvements in the evaluated downstream tasks.
Our experiments establish the practical value of this approach for large-scale open-vocabulary mapping, and the robot experiments demonstrate onboard deployment with reasonable latency. Through the release of model checkpoints, training recipes, and the data-generation pipeline, we also aim to make this family of methods easier to reproduce, use, and extend. Together, despite its conceptual similarities with EPM, we believe SceneLM makes a significant contribution, and provides further insights, to advancing this research field.