acceptodds
Withdrawn from ICLR 2027

SpatialMem: Object-Level 3D Memory Boosts Spatial Understanding from Monocular Videos

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatial understanding from monocular video requires locating objects in 3D and recognizing the same object across different views. Predicted depth and camera poses place observations in a common 3D coordinate system, but connected regions in the reconstruction do not always correspond to individual objects. Reconstruction artifacts may join geometry from neighboring objects, while occlusion may leave a single object represented by disconnected fragments. These ambiguities can compromise the semantic and geometric consistency of scene representations, undermining spatial understanding. To this end, we introduce SpatialMem, a framework that constructs reusable object-level 3D memory from monocular video by combining predicted geometry with cross-view visual evidence. The system associates semantic masks across views through the reconstructed geometry and uses visual evidence to determine which observations belong to the same object. It stores each instance with its observed geometry and links to supporting views. The memory supports both read-only spatial programs and a lightweight latent-injection interface to a frozen vision-language model. Using up to 300 sampled RGB frames per video for memory construction, SpatialMem achieves a seven-task average of 65.3% on ReVSI, the highest among the compared methods. In exploratory evaluations, latent features extracted from this memory also improve exact-match accuracy from 53.2% to 58.2% on SQA3D and from 28.6% to 30.6% on ScanQA. Our code will be released at https://anonymous.4open.science/r/SpatialMem-7E3D/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial understanding, object-level 3D memory, monocular video, object identity, multi-view association, spatial reasoning, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BRaPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpatialMem constructs object-level 3D memory from monocular videos using predicted geometry, segmentation masks, and identity refinement. The memory supports program-based spatial reasoning and learned VLM readout. It achieves 65.3% on ReVSI and reports exploratory gains on SQA3D and ScanQA.

Critical strengths
  • The method addresses merged objects and fragmented identities, with substantial counting gains supporting the refinement pipeline.

  • Competitive ReVSI results are accompanied by useful diagnostics connecting memory-construction errors to downstream counting and geometric-query failures.

Critical weaknesses
  • Observation and computation budgets are not matched: SpatialMem constructs memory from up to 300 frames, while many baselines use 32–64. Controlled comparisons are needed to isolate the benefit of identity refinement from additional observations and processing.

  • Calibration data overlap with evaluation scenes: Appendix B.3 reports 11 overlaps with SQA3D test and 18 with ScanQA validation. Although disclosed as exploratory, the learned-readout results require training-only calibration to establish held-out generalization. This concern does not directly invalidate the ReVSI results.

Questions for authors
  • Does the ReVSI advantage persist at a matched frame budget, such as 64 frames? What are the construction cost, VLM-call counts, and amortized query latency?

  • How do the learned-readout results change after retraining with calibration scenes strictly disjoint from evaluation scenes?

  • How often do queries use instance-level operations versus category-level fallbacks? Can direct instance-quality metrics or a matched alternative-association baseline substantiate the identity-refinement contribution?

Reviewer_GCqyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SpatialMem, a method to construct a object-level 3D scene for VLM spatial QA. It uses DA3 and SAM3 to form 3D candidates, and refine and verify them through rule-based and VLM-based determination to get instances in voxel representation. It also proposes to encode the predicted scene information using a vector encoding and show positive results on SQA3D and ScanQA. This method improves spatial QA results on ReVSI, showing the effectiveness of the proposed spatial memory.

Critical strengths
  1. The motivation of this paper is well understood. Agentic spatial reasnoning, especially by constructing a explicit memory, is getting a lot attention recently, and the proposed voxel representation is intuitively clear.
  2. This method achieves high scores on ReVSI.
  3. The paper examines how splitting can increase overcounting, how verification can remove geometric references required by later queries, and how merging trades reduced overcounting against increased undercounting. These analyses are more informative than aggregate scores alone.
Critical weaknesses
  1. The proposed method uses up to 300 frames to construct spatial memory, which is substantially more than several baselines. How would it perform if restricted to 32 or 64 frames, as used by those methods?
  2. Erosion is used in Table 5. However, this operation can disconnect thin structures and make small regions disappear. How are these effects addressed?
  3. There are many hardcoded thresholds that need further explanation and sensitivity analysis, including the minimum of 50 observation records per component per frame, 30 records per mask, five distinct voxels after erosion, and other thresholds in Table 5 and the appendix. How were these values selected?
  4. An incorrect SAM3 mask can act as a bridge connecting two distinct instances in the mask-support graph. How is this avoided or corrected?
  5. The motivation for the row encoder is unclear. Does this way of using spatial memory produce better results than directly querying the same memory through tools?
Questions for authors

The questions are included in the weaknesses above. Given the current concerns about the fairness of the comparisons, robustness of the pipeline, and motivation for the feature-injection branch, my tentative rating is 2. Hovever, clarifications and additional evidence that convincingly address these concerns could lead me to raise my score.

Reviewer_LhcfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper aims to build a spatial object centric memory of the scene and supply this memory to a VLM for downstream applications like VQA. The method proceeds as follows: a) first, a 3D object memory is build by using off the shelf models like SAM-3 and DA-3: SAM-3 provides segmentation masks per frame, which are lifted to 3D using DA-3; then an intial voting generates initial set of object candidates b) Next, the paper proposes a hand-designed refiner module which resolves errorneous merges of object masks using graph alignment with the corresponding 2D masks, and further checks with a VLM; similar hand-designed modules also help in merging objects that were erroneously seperated in 3D. b) Next, this object memory is used in two ways: i) by parsing the question into programs which directly operate over these object memories (in text space); ii) by allowing the VLM to cross-attend to these object memories in object centric way.

The paper evaluates its method on ReVSI, ScanQA, and SQA3D benchmarks and obtains superior performance over prior methods.

Critical strengths
  • The significant improvement on ReVSI and ScanQA/SQA3D benchmarks by the proposed method
Critical weaknesses
  • I have significant concerns over this paper's novelty and how it covers the relevant literature (both are connected).

a) One contribution of this work is in building 3D object-centric memory. There is a long line of work which builds such 3D object memory which includes using pre-trained 3D object detectors (3DETR on posed-RGBD, ImVoxelNet and follow-ups on RGB images); or methods that group and build these proposals using 2D open-vocabulary methods (like OpenMask3D which the paper also cites but does not really contrast on how/why their method is different/better), Open3DIS), methods which explicitly build object graphs for memory like 3DMem (), DynaMem, EPM (https://openreview.net/pdf?id=79BOATBal9) and many others.

b) Connecting this spatial memory to LLMs which again has a long line of work that this paper misses. For example, original methods of Referit-3D use object centric representations for 3D grounding; ChatScene, LEO are some recent works which connect object centric representations to 3D VLMs. Model's like LEFT (https://arxiv.org/abs/2310.16035) parse questions into programs which utilize object detections as part of the code execution [which in turn follows a long literature on Neuro-symbolic concept learner: https://arxiv.org/abs/1904.12584]

I think how a paper is positioned around existing literature is important to clarify, which the current related work section fails to do (but rather reads like a collection of prior works cited without explicit comparisons and contrasts)

  • Table-1 has some inconsistent results: the proposed model seems to use upto 300 frames, while baselines seem to use less. Have the authors tried to supply the baselines with more frames as well?

  • The experimental section is very hard to follow; statements headings like 'Split provides finer candidates, not final identities'; 'Verify affects which spatial references survive' is hard for me to infer and understand the significance of them.

  • The results on ScanQA and SQA3D look unfair as it seems that the base Qwen model is frozen while the proposed method is fine-tuned. The baseline qwen model should use similar training for fair comparisons. The inconsistency between frame counts reappear in these experiments (24 frames for the baseline, upto 300 frames for the proposed method)

  • I did not understand what this section meant as it seems to introduce new terms: "Current-scene content matters. The augmented model adds both scene-table information and a trained readout to the frozen backbone. Table 4 tests the contribution of scene content by fixing the readout and replacing only its scene-table input. With normal video, the current table outperforms all replacements. Its EM gains over the constant table are 6.0 percentage points on SQA3D and 1.6 on ScanQA, supporting a contribution from scene-specific content. With black VLM frames and video-derived memory unchanged, the current table again outperforms every replacement. However, scores remain below those with normal video (Appendix C.1)."

What is scene-table? What is current table? What is constant table? What is black VLM frames?

Questions for authors

In its current form this paper suffers from severe limitations in scoping it with respect to existing literature as well as writing of the experimental section. I do not think a rebuttal will help change my ratings, hence I am leaving this section blank.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.