acceptodds
Under review as a conference paper at ICLR 2027

SpatialBrain: Spatial Intelligence Growing from Experience

12 views

Abstract

Spatial reasoning requires more than access to geometric tools: a reasoner must decide when direct observation is sufficient and when to reconstruct, measure, or change reference frames. We present SpatialBrain, a framework that retains ex- perience about these decisions while keeping its vision-language backbone fixed. The key distinction is between evidence about a scene and experience about how to reason with that evidence. 3DFusionMemory fuses current-scene observations into queryable spatial relations. A Spatial Skill Library provides reusable tool pro- cedures, and Spatial Cognition associates task conditions with experience about when and how to use memory and skills. Completed trajectories are abstracted into process records and consolidated between task blocks, allowing later deci- sions to draw on earlier execution without transferring scene coordinates or refer- ence answers. Across five spatial-reasoning benchmarks, SpatialBrain improves on direct answers from the same backbone on four and ties on the fifth. Com- ponent comparisons, experience-coverage curves, and traced resource decisions characterize the framework’s behavior and distinguish overall task performance from the effects of experience reuse.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Intelligence, Agent, Self-evolving, Spatial reasoning, Multimodal agents, 3D working memory, tool use

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EgcjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpatialBrain uses a fixed vision-language model, spatial tools, and past experience to answer spatial questions. It has three parts: 3DFusionMemory for the current scene, a Spatial Skill Library for reusable procedures, and Spatial Cognition for choosing how to reason and which resources to use. Experience is updated between task blocks without reference answers. The full system outperforms direct answers from the same model on four of five benchmarks and ties on the fifth.

Critical strengths

S1. Using past experience to guide spatial reasoning and tool use is an interesting problem. Separating current-scene evidence from reusable experience makes sense.

S2. The experiments cover five benchmarks. The full system improves over direct VLM inference on four of them, with a particularly large gain on MindCube.

Critical weaknesses

W1. I could not find key inference settings, such as temperature and token limits, in the experimental setup or Appendix A. This raises concerns about reproducibility. It is also unclear whether the reported results come from a single run or several runs, and how much they vary between runs. Please provide the inference settings, including any defaults used, and report the number of runs and the variation in scores.

W2. The main experiments evaluate SpatialBrain only with GPT-5.6-sol. Including other models as baselines does not show whether the framework works with those models. I would like to see SpatialBrain tested with other backbones, including open-source models with different capability levels, each compared with its own direct-answer baseline. This would help show whether the framework is useful beyond this particular model.

W3. Which backbone is used for Skill-3D in Table 1? The table does not specify this, while SpatialBrain uses GPT-5.6-sol. My understanding is that the original Skill-3D paper reports results with GPT-5.4. Were the results here obtained with GPT-5.4 or rerun with GPT-5.6-sol? Since both methods use tools, comparing them with different backbones makes it hard to tell whether the gap comes from the frameworks or the models. Please clarify the setup and compare them using the same backbone, with comparable tool access and inference budgets.

Questions for authors

Q1. What inference settings were used, including temperature, token limits, and any defaults? How many runs were performed, and how much do the scores vary between runs?

Q2. Can you test SpatialBrain with other backbones, including an open-source model, and compare each with its own direct-answer baseline? This would help show whether it works across models with different capability levels.

Q3. Which backbone was used for the Skill-3D results in Table 1? Were these results taken from the original paper or rerun? Can you compare Skill-3D and SpatialBrain using the same backbone, with comparable tool access and inference budgets?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Egcj: Reproducibility, Cross-Backbone Evaluation, and Skill-3D Comparisons (part 1)

Response overview. Thank you for your detailed suggestions on reproducibility, applicability across backbones, and fairness of baseline comparisons. In response, we have added verified inference settings, scores and variation from repeated runs with the same task order, and a comparison of the open-source Qwen2.5-VL-32B-Instruct with its own Direct baseline. On MindCube, Frozen and Update improve accuracy over this model's own Direct baseline by 6.00 and 2.10 pp, respectively. We have also revisited the provenance and comparison conditions of the Skill-3D results. Below, we describe the confirmed settings, experimental results, and conditions that have not yet been fully aligned.

W1 / Q1: What inference settings were used, including temperature, token limits, and any defaults? How many runs were performed, and how much do the scores vary between runs?

The inference settings for the new experiments using the recovered setup are as follows.

SettingNew GPT-5.6-sol comparisons
temperature, top-pReasoning and code generation both use the API service's default sampling settings; these parameters are not explicitly specified in requests. All comparison conditions with the same backbone use consistent settings
Output limit per callUp to 7168 tokens for reasoning and 4096 tokens for code generation, both reduced according to the estimated input budget
Client-side context-length estimate128000 tokens for reasoning; 32768 tokens for code generation
sampling seed, reasoning effortNot explicitly sent in requests; the task-order seed is not the sampling seed
Rounds and time limitsAt most 8 Solver rounds; 1200 seconds per sample, 120 seconds for failure cleanup, and 300 seconds per tool call
Retries0 SDK retries; 240 seconds per planning-role attempt, 720 seconds total including retries, and at most 5 retries; terminal errors may stop execution earlier
Cumulative budgetNo separate hard limit on cumulative tokens per question; per-call limits do not imply equal total computation
Image processing and task organizationLLM image inputs are adjusted according to EXIF, converted to RGB, and resized to a maximum long side of 2048; tasks are grouped by scene, with block size 64 and task-order seed 20261113
Policy admissionMinimum support 8, risk tolerance 0.25, and minimum cost-improvement threshold 0.05; the risk bound concerns process violations, not a guarantee of true accuracy
Experience retrieval and explorationMinimum experience support 3; retrieval top-k 2 per question with a 256-token text budget; at most 4 canary explorations per block

We have added four complete MindCube runs with the same task order. Frozen retains the initial skills and scene memory while freezing cross-task experience; Update is the framework with online updates. The table reports accuracy, with the sample standard deviation across the four runs following ±; units are percentages or percentage points as appropriate:

Dataset / conditionRun 1Run 2Run 3Run 4Mean ± sample standard deviation
MindCube Frozen67.9066.7643.8169.5262.00 ± 12.18
MindCube Update68.2968.4844.0071.8163.14 ± 12.86
MindCube paired difference Update−Frozen+0.38+1.71+0.19+2.29+1.14 ± 1.02

Each run evaluates all 1050 MindCube questions, counting failures and empty answers as incorrect. Standard deviations are calculated from the four actual run results.

Additional reruns with a different task order. The following table reports complete evaluations under the specified new order (stream-order seed = 20261115). Frozen and Update use identical task membership and order within each dataset.

DatasetQuestions per conditionFrozen correct count (accuracy)Update correct count (accuracy)Update−Frozen
MindCube1050754 (71.81%)759 (72.29%)+0.48 pp
MMSI1000188 (18.80%)212 (21.20%)+2.40 pp
AuthorsOpenReview ↗

Response to Reviewer Egcj: Reproducibility, Cross-Backbone Evaluation, and Skill-3D Comparisons (part 2)

W2 / Q2: Can you test SpatialBrain with other backbones, including an open-source model, and compare each with its own direct-answer baseline?

We have added a Direct/Frozen/Update comparison on MindCube using the open-source Qwen2.5-VL-32B-Instruct. Direct denotes the model's own direct answers; Frozen and Update denote SpatialBrain with frozen and updated experience, respectively.

DatasetDirectSpatialBrain FrozenSpatialBrain UpdateFrozen−DirectUpdate−Direct
MindCube392/1050 (37.33%)455/1050 (43.33%)414/1050 (39.43%)+6.00 pp+2.10 pp

The model is fixed to revision 7cfb30d71a1f4f49a57592323337a4a4727301da; reasoning temperature/top-p are 0.6/0.95, and code-generation temperature/top-p are 0.0/0.95. Output limits match the role-specific limits above, and the service uses BF16.

On MindCube, Qwen2.5-VL-32B-Instruct with Frozen and Update improves over its own Direct baseline by 6.00 pp and 2.10 pp, respectively.

On MMSI, four official subsets also show positive differences for the framework over Qwen-32B's own Direct baseline. Camera motion and camera–camera positional relations show larger differences. The following table reports correct counts over all members of the four categories, comparing Frozen and Update with Direct from the same model:

Official question typeQuestionsDirectFrozenUpdateFrozen − DirectUpdate − Direct
Attribute (Meas.)64242535+1.56 pp+17.19 pp
Motion (Cam.)74153738+29.73 pp+31.08 pp
Positional Relationship (Cam.–Cam.)93183731+20.43 pp+13.98 pp
Positional Relationship (Cam.–Obj.)86172118+4.65 pp+1.16 pp

These four categories are post hoc breakdowns of the same complete MMSI evaluation, and each includes all members of the corresponding official question type.

GPT-4o shows a positive subset result: given the time constraints of the rebuttal period, we evaluated the same fixed hash-sampled subset of 128 MMSI questions. SpatialBrain answers 2 more questions correctly than the model's own Direct baseline, an accuracy difference of +1.56 pp.

Highlighted API backboneSubset sizeSpatialBrain−own Direct
GPT-4o128+1.56 pp

The GPT-4o comparison above uses a fixed 128-question subset, counting failures and empty answers as incorrect.

W3 / Q3: Which backbone was used for the Skill-3D results in Table 1? Were these results taken from the original paper or rerun? Can you compare Skill-3D and SpatialBrain using the same backbone, with comparable tool access and inference budgets?

The local adaptation used GPT-5.5, the public tool stack, and a static seed library; the values in all five columns match the saved historical summary. The distinctions are as follows:

Result sourceConfirmed informationComparison scope
Historical local adaptationGPT-5.5 static-seed adaptationCannot be described as a complete Skill-3D reproduction or as having all historical runtime parameters verified
Current SpatialBrain conditions with additional computationScored results for all four backbones on public PR365Additional computation and recovery conditions are reported separately and do not replace the original scores

The following table lists SpatialBrain scores with GPT-5.4 and Gemini 3 Flash alongside the Skill-3D paper's reported scores for the corresponding backbones:

SpatialBrain achieves 60.82% accuracy with GPT-5.4 and 55.07% with Gemini 3 Flash, exceeding the Skill-3D paper's reported scores for the corresponding backbones by 0.42 and 0.27 percentage points, respectively.

BackboneSpatialBrainSkill-3D (paper-reported score)Difference
GPT-5.460.82%60.40%+0.42 pp
Gemini 3 Flash55.07%54.80%+0.27 pp

Thank you again for emphasizing experimental transparency and rigorous comparisons. Your suggestions have helped us report inference settings, variation across runs, and results across backbones more accurately, and clarify the provenance and comparison scope of the Skill-3D results. We hope these additions provide a clearer and more verifiable basis for assessing the paper's experimental evidence.

Area_Chair_vh8wOpenReview ↗

Feedback

Hi, the review deadline is approaching. Please take a look at the authors' rebuttal and post your feedback (and update your rating if appropriate) at your earliest convenience. Thank you! AC

Reviewer_A8QmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents SpatialBrain, a spatial-reasoning agent that combines current-scene evidence, reusable computational procedures, and cross-task experience while keeping its vision-language backbone fixed. Its three components are 3DFusionMemory, which integrates observations into queryable spatial relations; a Spatial Skill Library, which provides parameterized tool procedures; and Spatial Cognition, which uses accumulated experience to guide reasoning routes, memory access, and skill selection. Experience is updated between task blocks without reference answers or transferring scene-specific coordinates. Evaluation spans five spatial-reasoning benchmarks. Compared with direct answers from the same GPT-5.6-sol backbone, the complete framework improves on four benchmarks and ties on the fifth, increasing the unweighted average from 67.5 to 72.6, with the largest gain on MindCube. Component comparisons, coverage analyses, and execution traces examine resource use and experience reuse, although they do not establish an aggregate causal benefit from continual experience updates alone.

Critical strengths
  1. A clear separation between scene evidence, reusable procedures, and experience about resource use. SpatialBrain distinguishes what the current scene supports from how spatial computations are performed and when those computations should be invoked. This decomposition provides a coherent approach to cross-task adaptation without transferring scene-specific coordinates or answers. Its explicit, inspectable updates make the framework useful for studying how agents can adapt their reasoning strategies while keeping model weights fixed.
  2. Meaningful gains over a shared-backbone baseline. The complete framework improves over direct GPT-5.6-sol answers on four of five benchmarks and ties on the fifth, raising the average from 67.5 to 72.6. The improvement on MindCube—from 67.2 to 87.6—is particularly substantial. Although these results do not isolate the contribution of experience updates, they demonstrate the practical value of the integrated system. Execution traces additionally document how stored experience can alter resource decisions, giving concrete support to the proposed mechanism.
Critical weaknesses
  1. The central benefit of experience-driven adaptation is not established. The main results compare the complete framework with direct backbone answers, combining the effects of tools, initial skills, scene memory, and experience updates. They therefore do not establish whether accumulating experience improves performance over an otherwise identical system with frozen initial experience. Coverage curves and selected traces demonstrate accumulation and use, but not aggregate gains in accuracy or efficiency; the earlier retrieval experiment also reports no overall improvement. This is consequential because “spatial intelligence growing from experience” is the paper’s central contribution.
  2. Confounded component comparisons weaken attribution of the gains. Appendix A acknowledges that configuration changes and unanswered outputs affect the reported ablations. For example, the headline MindCube results suggest a large memory contribution, whereas the matched Full/no-Memory comparison yields only a 0.67-point difference. The earlier MMSI configurations were also not verified as complete component removals. Consequently, these experiments do not reliably establish the independent contributions of the proposed modules, leaving uncertainty about which mechanisms explain the complete system’s improvements.
Questions for authors
  1. Evidence for the central adaptation claim. Do you have a matched comparison between continual experience updates and frozen initial experience, holding the backbone, tools, seed skills, controller, and inference budget fixed? What aggregate accuracy and total-cost differences can be attributed specifically to experience updates, including the cost of consolidation? This concerns validation of the current contribution rather than a future extension.
  2. Clarification of component comparisons. Appendix A reports only a 0.67-point gain in the matched MindCube Full/no-Memory comparison, substantially smaller than the headline gap. Which configuration differences account for this discrepancy? Which reported comparisons isolate individual component effects, and how much of the remaining differences arises from unanswered outputs?
  3. Clarification of procedural learning. In the configuration underlying the main results, were any newly compiled skills subsequently executed on later tasks, or did adaptation primarily change routing and access to existing skills? Please distinguish newly generated procedures from stored procedural records and changes in skill availability. This distinction materially affects the scope of the claimed self-evolution.
4 comments
AuthorsOpenReview ↗

Response to Reviewer A8Qm: Evidence for Experience-Driven Adaptation: Frozen Controls and Experience Snapshots (part 1)

Response overview. Thank you for your careful assessment of experience-driven adaptation as the central contribution, particularly your questions about frozen-experience controls, performance gains, and attribution to specific mechanisms. We agree that enabling experience updates should be examined through a control that retains the same initial resources and controller. We have therefore added full-task-stream Frozen/Update comparisons and evaluations of experience snapshots. Across four complete MindCube runs with the same task order, the update increments are +0.38, +1.71, +0.19, and +2.29 pp, averaging +1.14 pp. Below, we use these results to explain the evidence for experience accumulation and use, the observed performance gains, and the scope of the attribution conclusions.

W1.1: The central benefit of experience-driven adaptation is not established.

This question is directly examined by comparing Update with Frozen. Frozen always reads the initial experience while retaining normal scene memory, initial skill invocation, and the controller. Update starts from the same initial state and updates experience between completed task blocks. The two conditions are matched in backbone service alias, primitive tools, task membership, task order, and declared call limits. Update−Frozen includes the complete update mechanism and its exploration overhead.

The following table reports repeated results for the full MindCube benchmark and selected official MMSI question types. MindCube includes the original three runs and one additional run, for four runs in total; MMSI includes three runs. Each cell gives “Frozen correct count → Update correct count (accuracy difference),” and the final column reports the mean and sample standard deviation of the differences across the corresponding number of runs.

Dataset / official subsetQuestions per runRun 1Run 2Run 3Run 4Mean difference ± sample standard deviation
MindCube full set1050713 → 717 (+0.38 pp)701 → 719 (+1.71 pp)460 → 462 (+0.19 pp)730 → 754 (+2.29 pp)+1.14 ± 1.02 pp
MMSI: Attribute (Meas.)6430 → 31 (+1.56 pp)24 → 29 (+7.81 pp)24 → 31 (+10.94 pp)—+6.77 ± 4.77 pp
MMSI: Positional Relationship (Obj.–Obj.)9437 → 42 (+5.32 pp)34 → 42 (+8.51 pp)38 → 41 (+3.19 pp)—+5.67 ± 2.68 pp
MMSI: MSR19887 → 83 (-2.02 pp)85 → 92 (+3.54 pp)75 → 78 (+1.52 pp)—+1.01 ± 2.81 pp
MMSI: Positional Relationship (Cam.–Reg.)8339 → 42 (+3.61 pp)43 → 46 (+3.61 pp)40 → 35 (-6.02 pp)—+0.40 ± 5.56 pp

Update exceeds Frozen in all four complete MindCube runs with the same task order, with an average improvement of 1.14 pp; the fourth run improves by 2.29 pp.

Additional reruns with a different task order. The following table reports complete evaluations under the specified new order (stream-order seed = 20261115). Frozen and Update use identical task membership and order within each dataset.

DatasetQuestions per conditionFrozen correct count (accuracy)Update correct count (accuracy)Update−Frozen
MindCube1050754 (71.81%)759 (72.29%)+0.48 pp
MMSI1000188 (18.80%)212 (21.20%)+2.40 pp

W1.2: Coverage curves and selected traces demonstrate accumulation and use, but not aggregate gains in accuracy or efficiency

We have added overall accuracy comparisons and use coverage and traces to explain how experience enters subsequent tasks. The mechanism evidence from the main MindCube Update stream is as follows:

Evidence levelVerified resultQuestion addressed
Traceable process1050 tasks, 19 state snapshots, and 1050 update eventsWhich experience state each task reads and which completed tasks contribute to an update
Experience consolidation4 textual resources in the final state, of which 2 become retrievableWhether execution records are consolidated into usable resources
Actual deliveryThe 2 retrievable experience resources are delivered across 125 subsequent tasks, yielding 155 non-empty deliveriesWhether experience actually enters the inputs to subsequent task solving
Existing skill execution199 tasks contain 200 skill-completion eventsWhether existing spatial computation procedures enter the execution path

These statistics show that the formation and use of experience are inspectable. The 200 executions are invocations of existing skills, not 200 new programs. We further evaluate the effect of experience use on accuracy through the Update/Frozen comparison in W1.1 and the fixed-suffix comparison in Q1.

AuthorsOpenReview ↗

Response to Reviewer A8Qm: Evidence for Experience-Driven Adaptation: Frozen Controls and Experience Snapshots (part 2)

W1.3: How should the early retrieval results be interpreted in relation to experience-driven adaptation?

Experience-driven adaptation includes early exploration and subsequent reuse. In the early stage, experience retrieval, applicability assessment, and necessary exploration incur additional computation. These investments support the discovery, organization, and invocation of reusable knowledge, and their value should be assessed in terms of subsequent task performance.

To directly examine the subsequent value of experience, we have added accuracy comparisons and experience-snapshot evaluations. With a common initial library and identical task order, Update exceeds Frozen in all four complete MindCube runs, averaging an improvement of 1.14 pp, with 2.29 pp in the fourth run (see W1.1). Across three fixed-suffix evaluations, both intermediate experience snapshots also exceed their corresponding initial snapshots (see Q1). Together with the 155 actual deliveries of experience across 125 subsequent tasks, these results provide evidence for experience-driven spatial reasoning under these experimental settings in terms of both accuracy and the process of actual reuse.

Q1: Evidence for the central adaptation claim.

We examine the role of experience at two levels: online updates and fixed experience states. The two comparisons are configured as follows:

QuestionControlInterpretation
Does enabling updates help?Complete Update/Frozen task streams with the same initial libraryThe observed increment from the overall online update process, including exploration and update overhead
Does the accumulated state have value for subsequent tasks?After a learning prefix, apply the initial, intermediate, and final experience snapshots to the same suffix, with updates frozen during evaluationOverall performance of a fixed experience state

Accuracy comparison across experience snapshots. MindCube uses a 267-question learning prefix and a disjoint 783-question suffix. We refer to the saved cross-task experience states as experience snapshots: the initial snapshot is the state before learning, intermediate snapshots 1 and 2 are two saved states during learning, and the final snapshot is the state after completing the learning prefix. These represent cross-task experience states, not reasoning rounds within a single question. Each state is fixed and independently evaluated on the same 783 questions, with experience updates disabled during evaluation. Each cell below gives “correct count / total questions (accuracy),” and every condition evaluates all 783 questions:

Learning replicateInitial experience snapshotIntermediate experience snapshot 1Intermediate experience snapshot 2Final experience snapshot
1531/783 (67.82%)566/783 (72.29%)568/783 (72.54%)526/783 (67.18%)
2467/783 (59.64%)568/783 (72.54%)562/783 (71.78%)481/783 (61.43%)
3330/783 (42.15%)583/783 (74.46%)586/783 (74.84%)449/783 (57.34%)

Both prespecified intermediate states exceed their respective initial experience snapshots: intermediate snapshot 1 yields +4.47 / +12.90 / +32.31 pp, and intermediate snapshot 2 yields +4.73 / +12.13 / +32.69 pp. These are favorable observations at intermediate stages, but the final state is lower than both intermediate states in all three replicates; final snapshot−initial snapshot is −0.64 / +1.79 / +15.20 pp.

The post hoc analysis of intermediate states on MMSI also observes positive differences in two complete replicates: intermediate snapshot 1−initial snapshot is +17.47 / +27.47 pp, and intermediate snapshot 2−initial snapshot is +16.27 / +29.47 pp. The final-state differences are −2.93 / +12.40 pp, both lower than the corresponding intermediate states.

AuthorsOpenReview ↗

Response to Reviewer A8Qm: Evidence for Experience-Driven Adaptation: Frozen Controls and Experience Snapshots (part 3)

Q2: Clarification of component comparisons.

The following table reports seven component conditions on MMSI from the same configuration batch, using Full as the reference. Each condition evaluates all 1000 questions. Differences are calculated as “ablated condition−Full,” so negative values indicate higher accuracy for Full.

Component condition in the same batchMMSI: correct count / 1000 (accuracy)MMSI: ablated condition−Full
Full448/1000 (44.80%)—
w/o Memory444/1000 (44.40%)−0.40 pp
w/o Skill433/1000 (43.30%)−1.50 pp
w/o Cognition447/1000 (44.70%)−0.10 pp
w/o Memory + Skill434/1000 (43.40%)−1.40 pp
w/o Memory + Cognition397/1000 (39.70%)−5.10 pp
w/o Skill + Cognition388/1000 (38.80%)−6.00 pp

On MMSI, Full exceeds the conditions jointly removing Memory+Cognition and Skill+Cognition by 5.10 and 6.00 pp, respectively; it exceeds the conditions individually removing Memory, Skill, and Cognition by 0.40, 1.50, and 0.10 pp, respectively. These results show the performance advantage of the complete component configuration on this dataset.

Component ablations examine system performance after modules are removed; the frozen-experience control in W1.1 retains the controller and initial resources and examines the increment from enabling online experience updates.

Q3: Clarification of procedural learning.

What has currently been verified is existing skill reuse and experience-guided resource use. We report three distinct processes:

ProcessMeaningCurrent evidence
Initial skill generation52 resources generated offline by an MLLM and fixed before use37 guides, 14 atomic skills, and 1 composite skill; 15 have execution contracts
Existing skill executionSelects an existing procedure, binds current inputs, and executes it through SkillToolHostC.2 and skill-completion records from the main task stream
Experience-guided usePast execution records influence subsequent prompts, resource selection, or the availability of existing skillsExperience states, delivery events, and the example in C.3

Code generated temporarily by PythonTool at runtime is not counted as a new persistent skill. We thus distinguish the procedural extension pathway supported by the system from the behavior already observed in experiments.

Thank you again for your detailed comments on the central contribution and its supporting evidence. They have helped us distinguish more precisely between experience formation and use, overall performance increments, and attribution to specific mechanisms. The additional controls above provide more direct evidence for the gains observed on MindCube.

Hi, the review deadline is approaching. Please take a look at the authors' rebuttal and post your feedback (and update your rating if appropriate) at your earliest convenience. Thank you! AC

Reviewer_7heZOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose SpatialBrain that takes the multi-view observations of the current scene together with the task query as input. For each task, it dynamically constructs and updates a scene-specific 3DFusionMemory that stores fused objects, spatial relations, camera states, and route information. A persistent Spatial Skill Library provides reusable parameterized procedures for invoking perception and geometry tools, while a persistent strategic memory records experience about when and how these resources should be used. After a block of tasks is completed, their execution trajectories are abstracted into process records and used to update the strategic experience and the procedural skill library. These updates then influence resource selection for later task blocks, while the concrete scene memory itself is rebuilt from scratch for each new task.

Critical strengths
  1. The paper presents an interesting and potentially valuable cognitive architecture for spatial agents by explicitly separating scene-specific evidence, reusable procedural skills, and cross-task strategic experience. This abstraction is more conceptually motivated than a conventional tool-orchestration pipeline.

  2. This paper is well written, and clearly introduce the whole pipeline that contains memory, skill library and cognition core mechanisms. Implementation details and parameters are introduced clearly. Moreover, related literatures are reviewed sufficiently.

Critical weaknesses
  1. Several mechanisms that are central to realizing evidence abstraction are underspecified, particularly cross-view object identity association in 3DFusionMemory (section 3.2), the construction of spatial relation edges, and the initialization (manually or MLLMs generated) of the skill library (C.2). The notation around the scene graph and memory is also unnecessarily confusing, it stated as scene graph in equation 1, but memory in equation 2.

  2. The proposed framework relies heavily on off-the-shelf perception, reconstruction, grounding, and geometry tools, so part of its spatial capability is inherited rather than introduced by the proposed method. The current experiments do not fully disentangle the benefit of this cognitive architecture from the benefit of having access to additional specialist tools.

Questions for authors
  1. Clarification for some important procedures in the proposed pipeline. Authors are expected to answer how cross-view object identities are associated before fusion in 3DFusionMemory, Appendix E only describes fusion after an object has already been matched. Similarly, the construction of spatial-relation edges is not sufficiently specified, including whether they are produced by deterministic geometric tools or inferred by the VLM, as well as the coordinate conventions, thresholds, and confidence criteria used. Finally, the initialization of the Spatial Skill Library is unclear: the paper states that seed skills initialize the library, but does not explain whether these skills are manually authored, MLLM-generated, or automatically synthesized from primitive tools. Appendix C.2 mainly demonstrates the reuse of existing skills rather than their construction.

  2. Thorough ablation analysis. The current ablations do not fully disentangle the contribution of the proposed cognitive architecture from the benefit of accessing specialist perception and geometry tools. A stronger controlled baseline should use the same backbone and exactly the same expert tools, but without the proposed 3DFusionMemory, Skill Library, and strategic-experience organization. Such an experiment would clarify whether the reported gains primarily arise from the “brain mechanism” itself or from the additional expert capabilities available to the agent.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7heZ: Clarifying the Architecture, Skill Initialization, and Shared-Tool Evaluation (part 1)

Response overview. Thank you for your careful review and constructive comments on the method definitions, implementation details, and reliance on tools. To address these points, we have added formal definitions of the three components and scene memory, clarified the construction of the scene graph and initial skills, and reported system comparisons with shared primitive tools. On the full MindCube benchmark, frozen SpatialBrain improves accuracy over Tool_only by 4.86 pp. Below, we address the method details, experimental evidence, and corresponding scope point by point.

W1: What does each of the three components do, and how do Equations 1 and 2 relate to the scene graph and memory?

The three components organize evidence about the current scene, reusable computational procedures, and decisions about resource use, respectively. We have clarified their roles and lifecycles in the local revision:

ComponentInformation stored or processedRole in reasoning
3DFusionMemoryObjects, spatial relations, camera states, and route history for the current taskIntegrates tool evidence and supports spatial queries and checks; constructed independently for each question
Spatial Skill LibraryPlanning guides and executable spatial computation procedures with applicability conditionsBinds current images, objects, and tool outputs to reusable procedures
Spatial CognitionExperience about resource selection and strategiesDetermines reasoning routes, memory access, skill selection, and verification steps

The rewritten Equation 1 is:

\mathcal H_b=(\mathcal L_b,\mathcal P_b).$$ Here, $i$ denotes a task in task block $b$, and $t$ denotes a within-task execution step. $\mathcal M_{i,t}$ is the task's scene memory, while $\mathcal L_b$ and $\mathcal P_b$ are the skill library and strategic store, respectively. Together, they form the experience snapshot $\mathcal H_b$ read by the task block. Equation 2 expands scene memory: $$\mathcal M_{i,t}=(\mathcal G_{i,t},\mathcal C_{i,t},\mathcal R_{i,t}),\qquad \mathcal G_{i,t}=(\mathcal V_{i,t},\mathcal E_{i,t}).$$ $\mathcal G$ is the scene graph comprising object nodes $\mathcal V$ and relation edges $\mathcal E$; $\mathcal C$ and $\mathcal R$ store camera states and route history, respectively. **The scene graph is part of scene memory, and scene memory is part of the system state.** Scene evidence is updated within a task, whereas cross-task experience is updated between completed task blocks. Tasks within the same block read a fixed experience snapshot, and scene coordinates remain independent across tasks. ### W2: Given its reliance on external tools, what is SpatialBrain's contribution? **Our research concerns effective tool use: selecting appropriate spatial computations, organizing evidence, and adapting subsequent resource use based on execution experience.** For example, relative-direction judgments require an explicit reference frame, multi-view evidence must be associated in a consistent coordinate system, and numerical tool outputs must be mapped to the spatial semantics of the question. We regard these inspectable decisions and computations as components of a spatial reasoning system. To examine the contribution of these organizational mechanisms beyond primitive tools, we added a Tool_only control. The additional contribution of experience updates is examined through a separate Update/Frozen comparison. These two comparisons address system organization and experience updates, respectively; the shared-tool results are reported in Q2.
AuthorsOpenReview ↗

Response to Reviewer 7heZ: Clarifying the Architecture, Skill Initialization, and Shared-Tool Evaluation (part 2)

Q1.1: How is the scene graph constructed, and what are the coordinate conventions, relation edges, and thresholds?

Due to space constraints, the original manuscript primarily illustrated the scene graph visually and did not sufficiently describe its construction rules. We have added definitions and reproducible worked examples to the revision while retaining the original figures.

StepCurrent implementation
Object associationApplies lower(strip(name)) to names and associates nodes with exactly matching normalized names; a new name creates a new node. There is currently no additional appearance-based or geometric instance disambiguation, so multiple instances with the same name may be merged
Center fusionComputes a weighted center from projected points and fuses it with the existing center using the mean point weight of that projection. Input points must already be in a common reconstruction coordinate system; fusion itself does not perform camera registration
Spatial relationsFor objects A→B, computes Δ=CB−CA\Delta=C_B-C_A, Euclidean distance, depth difference, and the signed dominant-axis direction
Reference frameThe reconstruction/world–camera-0 coordinate convention is +X right, +Y down, and +Z forward. When the source object's pose is available, RA⊤ΔR_A^\top\Delta transforms the displacement into the object reference frame; otherwise, the reconstruction reference frame is used
ScaleDistances are initially expressed in reconstruction units; metric distances require additional scale estimation

For a single projection, let cc be the weighted center and qq the mean point weight, with existing center CC and accumulated weight QQ. When the corresponding sums of weights are positive, fusion is given by:

C'=\frac{QC+qc}{Q+q},\qquad Q'=Q+q.$$ When the sum of projection weights is non-positive, the arithmetic mean of the points is used. If no existing center with positive weight is available, the current center is stored directly. The relation support score combines accumulated projection weights, support from distinct views, and directional dominance. Let $m_1,m_2$ be the largest and second-largest absolute displacement components, and let $\operatorname{clip}$ truncate values to $[0,1]$. Then: $$r=\frac{m_1-m_2}{\max(m_1,10^{-6})},\qquad s=\operatorname{clip}\\!\left[\min(\operatorname{clip}(Q_A),\operatorname{clip}(Q_B))\\,v\\,(0.5+0.5\operatorname{clip}(r))\right].$$ If both nodes have detections in at least two distinct views, $v=1$; otherwise, $v=0.7$. This is a **heuristic evidence support score**, without probabilistic calibration; repeated projections do not increase the number of independent views. Direction labels are determined by the dominant axis and its sign. Ties between absolute components are resolved in X, Y, Z order, and zero displacement follows the same classification rule without a separate category. ### Q1.2: Were the initial skills manually written or generated by an MLLM, and what was the process? **The initial skills were generated offline by an MLLM and fixed before each run.** The generation inputs comprised task-type descriptions, tool interfaces, and examples, producing skill descriptions and programs. A sample of the generated programs was run to check that the sampled programs were executable. | Stage | Inputs or outputs | Checks and use | |---|---|---| | Offline generation | Task types, tool interfaces, and examples → skill descriptions and programs | Examples included official evaluation questions, without reference answers | | Structured storage | `SKILL.md` stores instructions; `manifest.json` declares applicability conditions, required tools, and execution information | The initial library contains **37 planning guides, 14 atomic skills, and 1 composite skill** | | Loading and execution | 15 executable skills declare execution contracts | The registry checks entry points, tools, input constraints, and references to subskills; current task inputs are bound at execution time | | Runtime checks | Some templates support numerical or output plausibility checks | Controlled by configuration; this does not amount to offline correctness verification of all skills | For example, the relative camera rotation skill requires two ordered views and camera extrinsics. The system binds the reference and target cameras for the current task, invokes the reconstruction tool, and then executes the stored rotation computation function to return the direction, angle, and principal rotation axis. The skill thus specifies when it applies, what evidence it requires, how to compute the result, and what to return.
AuthorsOpenReview ↗

Response to Reviewer 7heZ: Clarifying the Architecture, Skill Initialization, and Shared-Tool Evaluation (part 3)

Q1.3: What do the examples in Appendix C.2 demonstrate?

C.2 shows how existing skills are executed in the current scene and how geometric evidence is used to answer questions. The computational paths of the two types of examples are as follows:

ExampleInputs and computationRole demonstrated
Relative camera rotationTwo ordered views → reconstructed camera poses → relative rotation, principal axis, and direction → answerReuses an existing rotation computation procedure to provide an inspectable geometric basis for direction judgments
Camera–object directionCurrent object position → transformation into or expression in the specified camera reference frame → displacement components → corresponding spatial directionConnects geometric quantities to the directional semantics of the question; coarse direction labels must be interpreted together with the full displacement

The skills in both examples already existed in the initial library. C.2 supports the interpretation of existing skill reuse; C.3 separately explains how experience changes skill availability and selection strategies.

Q2: Thorough ablation analysis.

We have completed a comparison between Tool_only (T) and frozen SpatialBrain (Frozen), sharing seven categories of primitive tools. Both use the same backbone service alias, raw images, and the same deployments and invocation implementations for detection, 3D reconstruction, object pose estimation, scale estimation, OCR, Python execution, and answer generation.

Tool_only uses a generic planning loop; Frozen adds scene memory, the initial skill library, and specialized control logic. Frozen's MemoryQuery and SkillToolHost are access interfaces for these components and are part of the intervention in this comparison.

Selected positive results. With shared primitive tools, higher accuracy is observed on the full MindCube benchmark and three official MMSI question subtypes. The subtype analysis is a post hoc descriptive analysis conducted after inspecting the overall results. Categories follow the benchmark definitions, and each category includes all corresponding members of that 1000-question evaluation.

Dataset / official subsetTool_onlyFrozen SpatialBrainFrozen−Tool_only
MindCube full set662/1050 (63.05%)713/1050 (67.90%)+4.86 pp
MindCube: rotation147/200 (73.50%)187/200 (93.50%)+20.00 pp
MindCube: among330/600 (55.00%)381/600 (63.50%)+8.50 pp
MMSI: Positional Relationship (Cam.–Obj.)45/86 (52.33%)56/86 (65.12%)+12.79 pp
MMSI: Positional Relationship (Cam.–Cam.)53/93 (56.99%)61/93 (65.59%)+8.60 pp
MMSI: Attribute (Appr.)33/66 (50.00%)37/66 (56.06%)+6.06 pp

These results provide concrete descriptive evidence that, with shared tool access, the combination of framework organization and initial resources can still benefit particular spatial tasks. The improvements are especially large for camera-related positional relations and rotation tasks, consistent with the reference-frame representation, geometric computation, and evidence organization addressed by the method.

Thank you again for your specific suggestions on the method presentation and experimental design. They have helped us clarify the roles of SpatialBrain's components, the source of its initial resources, and the experimental results with shared primitive tools. We hope these clarifications and additional evidence address your main concerns and make the method's contribution and scope clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.