acceptodds
Withdrawn from ICLR 2027

See Once, Ask Many: Object-Anchored Neural Memory for Multi-View Spatial Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Multi-view spatial reasoning requires consistent object identity and reference-frame reasoning across partial observations, which remain difficult for vision-language models (VLMs). Existing methods often use external 3D models to provide spatial features, which VLMs then combine with visual semantics to directly predict spatial answers. We introduce See Once, Ask Many (SOAM), which learns object-addressable neural scene memory within the VLM and uses it to compose question-specific programs for explicit geometric execution. Given sparse views, a single VLM performs neural scene compilation: it organizes cross-view observations into anonymous objects and binds their visual representations into a question-independent neural memory. Object identities provide shared addresses for semantic grounding, view-local localization, and geometric computation. The same VLM then interprets each question through this memory and composes a typed program, while a frozen geometric backend supplies measurements for execution. The resulting scene representation supports subsequent questions. Joint state–program supervision and reuse-aware policy optimization train the shared memory for observation fidelity and utility across questions, with execution feedback assigned to each program individually. SOAM-8B outperforms the strongest baseline on VIEW2SPACE-v1 by 6.96 points in grounding mIoU and 5.28 percentage points in MCQ accuracy, while achieving a macro score of 72.85 on SPAR-Bench. Ablations show higher task scores with object-anchored memory and full training than with their respective controls.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
spatial reasoning, vision-language models, multi-view reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_6h98OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors focus on multi-view spatial reasoning problem and propose a See Once, Ask Many (SOAM) framework. In this framework, the input images are first translated into reusable and object-anchored neural scene graph. Based on this scene graph, VLM will generate programs for each input question for explicit calculation of specific spatial properties. Experiments on VIEW2SPACE-v1 and SPAR-Bench show the effectiveness of this framework.

Critical strengths
  1. Instead of leveraging a neural network-based model as previous methods, the authors use coding agents to design specific programs to deal with multi-view spatial reasoning problems, which is more explainable and can take advantage of the outstanding coding ability of current agents.

  2. Once the scene graph is constructed, different questions could be answered within the same scene graph, which will increase the efficiency.

Critical weaknesses
  1. Because the program backend is based on external geometry models but their prediction can be noisy and incorrect, I wonder how do the authors deal with such problem.

  2. To my understanding, the whole framework is trained with labeled object states and task-specific output formats. Therefore, I wonder how could the training scale with more data and whether it will fail in open-ended scenes or naturally phrased questions.

Questions for authors
  1. The first step of the framework is detecting objects and clustering the detection of the same object. I wonder what tracking/detection model is leveraged and how does the framework perform when this step fails.
Reviewer_z7AZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SOAM, which constructs an object-anchored scene representation and neural memory from sparse multi-view images. Question-specific typed programs bind semantic roles to object identities and perform geometric reasoning through explicit execution. The method reports strong results on VIEW2SPACE-v1 and SPAR-Bench and reuses the scene representation across questions.

Critical strengths
  1. The separation of semantic binding from geometric computation is well motivated. The framework offers a concrete way to reuse object state, visual memory, and geometry for multiple questions about the same scene.
  2. The ablations provide useful evidence for object-anchored memory and explicit execution. Comparisons with global pooling and alternative output/execution designs, together with memory reassignment interventions, help explain the role of the proposed modules.
Critical weaknesses
  1. The practical efficiency advantage is not established by runtime measurements. Figure 4 uses an analytical cost model with specified startup and per-question costs, but their measurement basis is unclear. End-to-end measurements including scene construction, geometric estimation, and question answering are needed to assess when reuse actually becomes advantageous.
  2. The incremental value of the particular neural-memory construction remains unclear. Object-centric representations and region-feature aggregation have precedents; the more specific contribution lies in using the shared VLM's hidden features and jointly learning scene state and programs. Global-pooling ablations establish the value of object anchoring, but do not isolate the advantage of this feature source over an alternative object-level representation under the same downstream pipeline.

Overall, the framework and module ablations are promising, but the evidence for its specific contribution and practical efficiency is not yet sufficient for me to recommend acceptance. I lean toward a weak reject.

Questions for authors
  1. Please provide the basis for Figure 4's cost constants and end-to-end latency measurements as the number of questions per scene increases, using the same hardware and including scene construction and geometric estimation. Does the measured crossover agree with the analytical model?
  2. Could the authors clarify the contribution of the shared-VLM feature source beyond object anchoring? A comparison with an alternative object-level feature source, keeping object state, geometry, and execution fixed, would help isolate this choice.
  3. The PDF available to me refers to Appendix ?? and Appendix B, but ends at the references without those sections. Please indicate where the appendix is available and clarify the supervision construction, training schedule, and baseline comparison conditions.
Reviewer_4RzcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes See Once, Ask Many (SOAM), a framework for multi-view spatial reasoning that decouples question-independent scene understanding from question-conditioned reasoning. SOAM first uses a vision-language model (VLM) to construct an object-centric scene representation, where anonymous object identities are associated with cross-view bounding boxes and continuous neural memories pooled from visual hidden states. Given a question, the same VLM generates a typed program conditioned on the cached scene representation, while a frozen VGGT model provides geometric information for deterministic execution. The framework is trained through joint state–program supervision and reuse-aware policy optimization, aiming to preserve scene information while improving downstream reasoning utility.

The authors evaluate SOAM-4B and SOAM-8B on VIEW2SPACE-v1 and SPAR-Bench. SOAM-8B achieves 70.21% MCQ accuracy on VIEW2SPACE-v1 and a macro score of 72.85 on SPAR-Bench. Additional experiments investigate object-anchored memory, semantic binding, object retention, and different training objectives. The results suggest that object-level memory and the proposed training objective improve spatial reasoning performance. The paper also presents an analytical cost model to characterize the potential benefits of reusing scene representations across multiple questions.

Critical strengths
  1. The paper introduces a unified object-addressing mechanism that connects cross-view observations, continuous visual representations, and geometric measurements. The memory reassignment experiment provides meaningful evidence that downstream semantic selection depends on the correspondence between object identities and their associated neural memories. In particular, the substantial degradation in MEMBERS set-F1 after memory reassignment supports the importance of correct object-memory binding. This provides a reasonable empirical foundation for the proposed representation.

  2. The proposed reuse-aware objective addresses an important limitation of task-driven scene representations: the tendency to preserve only information relevant to current questions. The ablation results show that incorporating observation fidelity improves both task performance and the recall of unqueried objects. In particular, the improvement in unqueried-object recall from 68.90% to 84.10% compared with task-reward-only training provides useful evidence for the proposed training strategy.

Critical weaknesses
  1. SOAM primarily integrates object detection, bounding-box-based visual feature pooling, structured program generation, and external geometric execution. In particular, Equations (2) and (3) essentially perform weighted pooling of object-region features and feature injection into object tokens, without introducing a fundamentally new mechanism for cross-view identity alignment or object-level representation learning. The current ablations only demonstrate that object-level features outperform global pooling, but do not establish superiority over simpler alternatives such as object queries or cross-view attention-based aggregation. Consequently, the central methodological contribution remains insufficiently substantiated.

  2. The observation fidelity term in Equation (6) primarily relies on bounding-box IoU and object matching, measuring object localization and preservation rather than whether the neural memories actually retain semantic and identity information. Therefore, this metric could remain high even if important appearance features are lost or different instances become semantically confused. Similarly, the reported 84.10% recall of unqueried objects only demonstrates that objects remain detectable, not that their associated memories retain sufficient semantic information for subsequent recognition and reasoning. This leaves the central claim of reusable object-centric semantic memory without direct empirical support.

  3. The RL training samples only two questions per scene, and the experiments do not adequately demonstrate whether the cached scene representation can support out-of-distribution or previously unseen questions. Furthermore, Figure 4 presents an analytical cost comparison based on predefined cost constants rather than actual measurements of end-to-end latency, GPU memory consumption, or throughput. Consequently, the claimed break-even point of eight questions is a theoretical result under specific assumptions rather than an experimentally established efficiency advantage. Overall, the paper does not sufficiently demonstrate its central claims regarding cross-question generalization and practical inference efficiency.

Questions for authors

See Critical Weaknesses

Reviewer_aBwrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SOAM for multi-view spatial reasoning. It first organizes a multi-view scene into a question-independent object-level state. For each question, it generates a typed program over this shared state and executes the program using geometry provided by a frozen model to obtain the answer. Training uses SFT followed by RL. Experiments on VIEW2SPACE-v1 and SPAR-Bench show improvements.

Critical strengths

The problem setting is meaningful. Repeating cross-view object correspondence for every question about the same multi-view scene is wasteful. Explicitly modeling scene understanding once and reusing it across questions is a reasonable direction. The reuse-aware training is interesting. Table 4 and Figure 3(c) show that training with task rewards alone lowers observation fidelity and reduces recall for unqueried objects, whereas adding observation fidelity restores both. This observation is also useful for multi-question VQA and agent memory design beyond this particular system.

Critical weaknesses

The paper does not make its core novelty clear enough. It combines scene reuse, object-level neural memory, separating semantic grounding from geometric computation, program generation, and training design. However, it does not clearly explain what the key new mechanism is relative to existing object-level memory and program execution methods. Table 2 and Figure 3(b) demonstrate the role of object-level memory and its binding, but these results are still insufficient to establish which gains reflect the core contribution that distinguishes this work from prior methods. Multi-instance grounding remains weak. In Table 3, exact-set success for 4–8 targets improves from 30.77% to 38.46%, but recovering all targets remains difficult. The action set and grammar are also predefined, and the current experiments do not sufficiently establish what kinds of questions this interface supports or whether it generalizes to unseen combinations of operations.

Questions for authors

What is the main novelty of this paper? Compared with existing object-level memory and program execution methods, what key new mechanism does SOAM introduce? Table 2 and Figure 3(b) support the role of object-level memory, but which experiment most directly supports the distinct contribution of this work over prior methods? How can the benefits of cross-question feedback be separated from those of scene completeness supervision? In Table 4, how exactly is “one-question state credit” implemented? Does it match the full objective in the number of questions, sampling count, and compute budget? Can the current results establish the separate contributions of these two designs? Figure 4 already includes scene construction, geometry construction, and program execution costs in its analytical model, but how are the cost constants chosen? Please report measured setup time and per-question time on the same hardware, and verify how many questions are actually needed for reuse to become more efficient than the shared-KV approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.