acceptodds
Under review as a conference paper at ICLR 2027

Spatial-JARVIS: A Meta-Level Spatial Cognitive Regulator for Interactive Spatial Agents

Abstract

Multimodal large language models (MLLMs) have made significant progress in passive spatial reasoning, yet they still face substantial challenges in long-horizon active spatial interaction tasks. During such interactions, inaccurate beliefs about the environment and task progress can lead to erroneous decisions and persist across subsequent steps, ultimately resulting in task failure. We attribute these failures to insufficient spatial metacognition, i.e., the ability to monitor and regulate evolving cognitive states and decision-making processes. To address this limitation, we propose Spatial-JARVIS, a Meta-Level Regulator decoupled from the policy model. At each interaction step, Spatial-JARVIS monitors the agent's inferred cognitive state and evaluates the proposed decision before execution, generating corrective feedback to guide the policy model in revising its mistaken beliefs and adjusting its subsequent behavior. To learn this capability, we construct the MetaSpatial-35K dataset and adopt a two-stage training strategy consisting of SFT and MC-GRPO. We further introduce MetaSpatial-Bench, a static benchmark designed to directly evaluate models' monitoring and correction capabilities in spatial interaction. Experimental results show that Spatial-JARVIS outperforms the strongest compared MLLMs by 14.6% and 13.3% in monitoring and control accuracy on MetaSpatial-Bench, respectively. Moreover, Spatial-JARVIS improves the task success rates of six different policy models by an average of 7.7% on SpatialWorld, while reducing ineffective exploration and erroneous operations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal large language model, interactive spatial intelligence, metacognitive

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ehGrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Spatial-JARVIS, a spatial regulator that operates independently of the interaction policy. Before an action is executed, it evaluates the proposed decision and provides action corrections and belief-revision feedback to guide current and subsequent decisions. The authors construct MetaSpatial-35K using simulator states, oracle trajectories, equivalent-action sets, and model-generated feedback, and train the regulator through SFT followed by MC-GRPO. They also introduce MetaSpatial-Bench for static evaluation. Experiments report improvements in monitoring and correction accuracy, an average task success rate gain of approximately 7.7 percentage points across six policies on SpatialWorld, and additional evaluations across environments and regulator sizes.

Critical strengths

Online improvements across multiple policies support the practical value of the approach. The evaluation goes beyond the authors’ static benchmark and includes six policies on SpatialWorld, together with transfer to EmbodiedBench-Nav. For example, Spatial-JARVIS improves Qwen3.5-27B’s success rate on SpatialWorld from 5.6% to 19.0%. These results provide meaningful evidence that the regulator complements the capabilities of the underlying policy.

The data construction and component analyses address relevant design choices. Equivalent-action sets avoid treating a single oracle action as the only valid decision, while policy-generated errors provide more realistic negative examples than random actions. Comparisons involving belief revision, general-purpose regulators, and a smaller regulator further support the value of specialized regulator training.

Critical weaknesses

The comparison with direct policy training does not fully isolate the benefit of the regulator architecture. The direct policy baseline primarily receives SFT, whereas Spatial-JARVIS receives both SFT and MC-GRPO and introduces additional model calls at inference time. Consequently, the comparison does not adequately separate the contributions of the regulatory structure, additional training, and additional inference resources. This limits the conclusion that training a regulator is more effective than directly improving the policy.

Supervision derived from privileged simulator states may not always be justified by the agent’s observations. Action validity and belief corrections are constructed using global simulator information, whereas the deployed regulator receives partial observations and interaction history. A reasonable exploratory action could be labeled incorrect because it does not advance the oracle’s current subgoal, and a correction could contain facts that cannot be inferred from the available observations. Scene and task separation addresses data overlap, but does not resolve this potential mismatch between the supervision and the information available to the regulator.

Questions for authors

Do you have a direct policy SFT+RL baseline using the same source data and comparable training compute, or a policy self-correction baseline with a matched inference-call budget? If so, do these comparisons still support the conclusion that training the regulator is more effective?

How do you ensure that negative actions are unreasonable given the agent’s observable information, rather than merely inconsistent with the globally informed reference trajectory? Have you audited whether belief-revision targets are inferable from the available observations and whether equivalent-action sets adequately cover reasonable exploratory actions?

Figure 1 appears to contain AI-generated visual artifacts. In the left panel, the top-down view and the perspective view appear spatially inconsistent; in the right panel, the cup appears to float rather than being held by the robot’s gripper. Could you clarify how this figure was produced? I recommend manually redrawing Figure 1 to ensure spatial consistency and physically plausible object interactions.

5 comments

W1 / Q1

We thank the reviewer for the suggestions on comparison fairness. We address this concern through comparisons with direct policy training and inference-time self-correction.

We agree that the original comparison did not fully isolate the contribution of additional reinforcement learning. We therefore added a direct policy baseline trained with SFT followed by GRPO, initialized from Qwen3.5-9B and using data from the same source and at a comparable scale. This two-stage training directly optimizes the policy’s action generation. Together with the existing results in Table 4, the comparison is as follows:

MethodDailyWorkEntertainTravelSocialOverall
Original policy, without additional training2.15.10.00.013.03.1
Direct policy training: SFT6.511.96.115.410.97.7
Direct policy training: SFT + RL8.315.313.60.04.39.2
Original policy + Spatial-JARVIS11.815.37.623.119.612.6

All values are task success rates (%). Extending direct policy training to SFT + RL increases its overall success rate to 9.2%, while Spatial-JARVIS achieves 12.6%. This comparison provides further evidence for the effectiveness of training an independent regulator when both approaches receive two-stage training.

Moreover, we clarify that Table 7 already includes a policy self-correction baseline, termed Self-Reflection. Using the same Qwen3.5-27B interaction policy, this baseline reflects on its preceding decision at each interaction step. It achieves a task success rate of 8.2%, compared with 19.0% for Spatial-JARVIS, a difference of 10.8 percentage points. The additional output-token consumption is comparable: both methods generate approximately 200 additional output tokens per interaction step on average.

These comparisons support the benefits of Spatial-JARVIS over both direct policy optimization with SFT + RL and policy self-reflection with comparable additional output-token consumption.

W2 / Q2

We thank the reviewer for highlighting the distinction between factual correctness and support from local observations. We have considered this issue in our data construction.

For the "privileged information and local observations", we construct decision-level samples using subtask-specific rules. During Search, when the target has not been observed and no directional cues are available, we allow exploratory rotations in different directions without requiring the agent to follow the oracle’s particular search direction based on the target’s true location. During Navigation, the target location has already been observed in the interaction history. The simulator’s reachability graph is used to verify progress toward this known target while retaining multiple actions that advance navigation. Thus, our construction accommodates diverse exploratory choices when the target location is unknown and evaluates navigation toward previously observed targets, without requiring access to unobserved target locations.

For the “belief revision”, we provide Qwen3.5-Plus with both the ground-truth simulator state and the observations and interaction history available to the agent. These inputs serve distinct purposes: privileged states verify environmental facts and identify discrepancies between the policy’s beliefs and the actual state, while observations and interaction history provide the evidence for corrective feedback. The model is instructed to determine which beliefs require revision based on existing observations, action records, and execution feedback, rather than directly incorporating hidden simulator information into the corrections. For example, observations following a grasp attempt or explicit failure feedback can support the correction that an object has not been picked up. Conversely, a location that has neither been observed nor can be inferred from the history should not be directly disclosed.

As discussed above, our subtask-specific rules aim to retain reasonable alternatives as comprehensively as possible: Search accommodates non-redundant exploration in different directions, Navigation retains multiple actions that advance the goal, and Interaction and Completion generally retain only the reference action. We further manually reviewed 50 randomly sampled examples and found no omitted reasonable alternatives in 47 cases, yielding a coverage-check pass rate of 94%. These results provide direct empirical support for the coverage of our equivalent-action sets.

Q3

We clarify that Figure 1 was manually composed, with AI-generated assets used only for local robot illustrations and scene details that are difficult to draw by hand. These assets were intended to make the illustration more intuitive; however, limitations in AI generation led to inaccuracies in spatial relationships and physical details. We sincerely thank the reviewer for highlighting this issue and will manually redraw the figure in the revised manuscript to ensure its accuracy.

Thanks for your detailed responses, which partially solve my concern.

One part of Q2 remains open for me. The 47/50 audit checks whether reasonable actions were omitted. Did you separately check whether the factual claims in the belief-revision feedback could be supported by the agent’s observations and action history alone, with the simulator state hidden from the annotator? If so, could you report the sample size and result?

A minor clarification: my figure comment referred to Figure 1 in the original submission, showing the inconsistent views on the left and the floating cup on the right. Could you confirm that this is the figure you plan to redraw?

Reviewer_o23aOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies persistent failures in multimodal large models during multi-step spatial interaction tasks caused by incorrect judgments about object locations, action execution results, or task progress. The authors propose Spatial-JARVIS, a supervisor model decoupled from the original policy model. It judges whether a candidate action is reasonable before execution and helps the policy model adjust its subsequent behavior through action correction and belief correction feedback.

Critical strengths

The research problem has clear practical value. The errors discussed in the introduction and Section 2.3 have effects across multiple steps. For example, after a failed grasp, the model continues as if the grasp had succeeded. Such problems cannot be identified solely through the final task success rate, which provides a reasonable motivation for studying monitoring and intervention before action execution.

The authors establish a complete research pipeline covering data generation, supervisor model training, and evaluation. The work is relatively solid.

Critical weaknesses

The data in this paper have explicit supervision labels, but their reliability depends on the completeness of the equivalent action sets and the error explanations generated by large models. In particular, it is necessary to verify whether actions labeled as unreasonable might simply be reasonable but suboptimal exploration choices. Direct validation of label quality is missing.

The paper studies error accumulation in multimodal agents during long-term spatial interactions. The authors categorize failures into task state misjudgment, subtask plan mismatch, spatial representation errors, and inaccurate judgments of action effects. These analyses are valuable, but “error types” and “root causes of errors” are not the same thing. The underlying causes of these problems are not analyzed in depth.

Questions for authors

In Figure 11, for example, the sample contains no action history, and the target laptop is not visible in the current view. The authors label “move forward” as unreasonable and use “turn left” as the corrective action, arguing that the agent should first scan other directions. When the target is not visible and no previous exploration information is available, why can moving forward be considered unreasonable rather than simply another exploration choice? This does not directly prove that the label is wrong, but the basis for judging the reasonableness of actions in such samples needs further explanation and validation.

Through experiments with different interaction budgets, belief correction ablations, and recovery rate analysis by category, the authors have provided valuable evidence about error persistence and correction. Could the authors further trace errors from their first occurrence through their subsequent propagation, and control the accuracy of historical information and states to verify which specific part of the process the supervisor improves? This could lead to valuable conclusions for the design of long-term spatial agent interactions.

4 comments

W1 / Q1

We thank the reviewer for the valuable comments. We clarify supervision quality from two aspects, equivalent-decision set construction and error-explanation generation, then provide direct human evaluation results.

We explicitly consider decision-set coverage during data construction by tailoring the rules to the characteristics of each subtask stage, retaining reasonable alternatives beyond the reference trajectory. Search is the most open-ended stage and therefore admits a broader range of exploratory alternatives. When the target has not been observed and no directional cues are available, rotations in different directions are accepted as reasonable exploratory actions, accommodating multiple choices under limited information. During Navigation, the target location has already been identified in the interaction history. We examine candidate movements in each direction using the simulator-provided reachability graph, retaining decisions that reduce the shortest-path distance to the target and including corresponding turning actions. This criterion uses traversable-path distance, allowing multiple movement choices and necessary obstacle avoidance. During Interaction and Completion, we retain the reference action without expanding the candidate set.

We carefully design the explanation-generation process to ground LLM outputs in verifiable evidence. Specifically, error explanations are conditioned on the interaction history, subtask stages verified against simulator states, and simulated outcomes of candidate decisions. They explain the current task progress, action consequences, and why a decision is unreasonable. As described in §3.2, another model further filters samples that contradict environment transitions, merely restate supervision labels, or contain low-quality reasoning, providing an additional quality check.

To directly assess action-set coverage and explanation reliability, we randomly sampled 50 examples and manually checked whether reasonable alternative actions were omitted and whether the error explanations were consistent with the interaction context and environment facts. The results are shown below:

Evaluation criterionExamples passing reviewPass rate
Sufficient coverage of equivalent-decision sets47/5094%
Correct error explanations49/5098%

Our pipeline incorporates multiple measures to ensure annotation quality. Although occasional errors and edge cases remain, the human evaluation supports its high action-set coverage and explanation reliability.

Regarding Figure 11. We apologize for the lack of clarity in the presentation. Turning left is only one acceptable decision in the equivalent-decision set; rotations in other directions are also reasonable exploratory choices when searching for the unseen target. In this particular scene, the laptop is absent from the current forward view, and moving forward does not reveal it. The forward decision is therefore labeled unreasonable because it does not provide useful exploration progress, rather than because it differs from the illustrated left-turn action.

W2 / Q2

An agent’s decision-making process during spatial interaction can be divided into three stages: observation (identifying relevant facts from current observations and interaction history), planning and reasoning (reasoning and planning based on dynamically updated spatial and task states), and decision-making (assessing whether the effects of a decision align with expectations). We have already presented the root causes of the four error types in the Introduction. These can be summarized as insufficient awareness of the agent’s own cognitive state during planning and decision-making, allowing erroneous beliefs or decisions to persistently influence subsequent decisions.

To identify which specific stages of the interaction process are improved, we selected 50 failed trajectories generated by Qwen3.5-27B and manually located the first identifiable erroneous judgment in each trajectory. Under the same environment state, observations, and interaction history, we separately provided three types of auxiliary information: (1) observational facts, detailing task-relevant facts directly visible in current and historical observations; (2) updated states, integrating current and historical evidence to clarify spatial relationships, object states, and task progress; and (3) action consequence analysis, specifying the execution preconditions and expected consequences of the original candidate action. We compared these interventions with monitoring and corrective feedback generated by Spatial-JARVIS and measured the correction rate for the first error in each trajectory.

Experimental conditionCorrection rate ↑
+ Observational facts6%
+ Updated states44%
+ Decision consequence analysis36%
+ Spatial-JARVIS32%

The results show that supplementing observational facts alone yields limited correction, whereas providing updated states or action consequence analysis corrects more errors. This indicates that Spatial-JARVIS primarily improves how the policy uses observations to update its state judgments and subsequently verify plans and actions, rather than merely identifying facts from observations.

We sincerely thank you again for your thoughtful review. We hope our clarifications and additional experiments have addressed your concerns. If any questions remain, we would greatly appreciate your feedback and would be happy to provide further clarification.

As the author–reviewer discussion period is closing soon, we would be grateful to hear whether our clarifications have addressed your concerns or whether any questions remain. Thank you again for your time and thoughtful feedback.

Reviewer_JnU1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on addressing the long-horizon spatial reasoning challenge faced by MLLMs. They target two common errors faced in long-horizon reasoning: 1) belief update, and 2) progress understanding. They introduce Sptial-JARVIS, a trained VLM critic that checks the policy output and provides prompt-conditioned correction to the policy. They create a dataset: MetaSpatial-35K, to train the Spatial-JARVIS. The training is performed in two stages: 1) SFT and 2) MC-GRPO. They further propose MetaSpatial-Bench, an evaluation benchmark to evaluate correction capabilities for spatial reasoning.

Critical strengths

Although the dataset construction pipeline is relatively standard, MetaSpatial-35K appears valuable given the reported performance gains. Its public release could be a useful contribution to the community.

Critical weaknesses
  1. The meta-cognition, as John H. Flavell says, is: "Metacognition refers to one's knowledge concerning one's own cognitive processes and products or anything related to them." Here, the authors are training a critic/correction module, so it would be much clearer to describe it as a VLM critic for action correction and "guidance", rather than framing the entire approach around metacognition. If the authors believe there is a meaningful distinction between the two (metacognition v.s. critic which includes correction and guidance on how and why to correct), I would request an experiment demonstrating this difference. (Happy to raise the score if clearly demonstrated)

  2. The novelty of the paper is limited. a) The problem statement is a fairly well-known, b) standard training implementation (SFT and GRPO), c) the dataset construction largely follows established approaches involving oracle trajectories, policy-generated errors, and corrective supervision [1]. Although action supervision, belief-revision targets, and hierarchical reward are useful, task-specific additions, their novelty relative to existing VLM critic-training methods remains limited.

[1] Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automation

  1. Lack of relevant works discussion: [1] Can MLLMs Find Their Way in a City? Exploring Emergent Navigation from Web-Scale Knowledge [2] CLEA: Closed-Loop Embodied Agent for Enhancing Task Execution in Dynamic Environments [3] Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs
Questions for authors

The paper reports aggregate results across task categories without separating performance by simulator. So, it's unclear how well Spatial-JARVIS generalizes to entirely unseen simulation environments.

7 comments

W1

We thank the reviewer for the comments. We further clarify our motivation for adopting a metacognitive perspective.

We first clarify the distinction and relationship between a Critic module and spatial metacognition. A Critic is a concrete approach to implementing evaluation and correction, whereas spatial metacognition provides a theoretical perspective on monitoring and regulating spatial cognition. A Critic typically evaluates and corrects an agent’s decisions and reasoning. Under our spatial metacognitive framework, our method retains these core Critic functions while explicitly incorporating the reliability and updating of the policy’s own spatial beliefs into its monitoring and regulation objectives.

This positioning is motivated by the characteristics of spatial interaction. Compared with Critics primarily targeting decision evaluation and correction in other agent tasks, such as GUI agents and tool-use agents, spatial interaction additionally requires agents to integrate observations across viewpoints and time, continuously updating spatial locations, object states, and task progress. These spatial beliefs provide a shared cognitive basis for multiple subsequent decisions. If they become inconsistent with observational evidence or are not updated promptly, their errors may persistently affect subsequent planning.

For example, suppose an agent mistakenly believes that a phone is on the table to its left, although earlier observations show that it is beside the bed. If a traditional Critic module is used solely to correct the current decision, correcting an erroneous turning decision may still leave the agent searching the table based on its mistaken belief. Spatial-JARVIS therefore addresses both decisions and their underlying spatial cognition: it evaluates and corrects decisions, verifies and revises the spatial beliefs supporting them, and retains corrective feedback in subsequent interaction memory and context. This enables the policy to adjust its planning and reduces repeated errors arising from the same cognitive misconception.

Table 7 already reports task success rates for decision evaluation alone (4.6%), decision evaluation with correction (16.1%), and the complete Spatial-JARVIS (19.0%). To further analyze the advantages of metacognitive regulation over a Critic alone, we plan to sample 50 failed trajectories and compare the following three settings at the same error points, focusing on error recurrence and its influence on subsequent decisions. We measure error recurrence rate and average number of affected steps. The former is the percentage of trajectories in which inappropriate decisions caused by the same erroneous belief occur again; the latter is the average number of subsequent inappropriate decision steps attributable to that belief per trajectory.

SettingError recurrence rate (%) ↓Average affected steps ↓
Critic: decision evaluation only88.05.20
Critic: decision evaluation and correction56.03.62
Spatial-JARVIS26.00.96

This analysis shows that decision correction reduces error recurrence and the number of subsequently affected steps. However, without explicit belief revision, the policy may continue to rely on erroneous spatial judgments and repeatedly produce errors stemming from the same belief. Introducing spatial belief regulation helps reduce the persistent propagation of these cognitive errors through subsequent interactions.

W2

We clarify that the main contribution of this work is not a particular algorithm, but a new metacognitive framework for spatial interaction. Existing approaches primarily focus on improving the decision-making capabilities of the policy model itself, while explicit mechanisms for monitoring and regulating evolving spatial cognitive states remain underexplored. We revisit this limitation from a metacognitive perspective and propose a two-level framework comprising task-level decision-making and meta-level regulation. Spatial-JARVIS, which is independent of the policy model, implements the monitoring and regulation of spatial cognition and decision-making.

We acknowledge that SFT and GRPO are established training methods, and that reference trajectories, policy-generated errors, and corrective feedback are also used in related data construction approaches, including GUI-Critic-R1. Our specific designs, however, are tailored to the characteristics of spatial interaction. We account for the coexistence of multiple reasonable decisions and the continuous evolution of spatial cognition and task states: equivalent-decision sets preserve reasonable alternatives, belief-revision supervision corrects mistaken spatial understanding and task-progress judgments, and hierarchical rewards jointly optimize decision monitoring and cognitive regulation. Together, these designs operationalize the metacognitive framework through data construction and model training.

Experiments demonstrate that the framework can be integrated with different policy models and improve their spatial interaction performance. We will also release MetaSpatial-35K, MetaSpatial-Bench, and the data construction pipeline, providing the community with data and tools for learning and evaluating monitoring and regulation capabilities in spatial interaction.

W3

We will expand the discussion of these related works in the revised manuscript.

Can MLLMs Find Their Way in a City? primarily investigates how pretrained knowledge and explicit cognitive maps can improve urban navigation. Our work instead proposes a metacognitive framework independent of the policy, learning to monitor the policy’s spatial judgments and improve subsequent interaction through decision correction and spatial belief regulation.

CLEA supports dynamic replanning through environmental memory and a Planner–Critic loop. Our distinction lies in establishing decision monitoring and spatial belief revision as explicit metacognitive learning objectives, using dedicated supervision to train a regulator to verify and correct the policy’s cognitive judgments.

Learning from Trials and Errors jointly updates the policy and internal reflection model through test-time training. Our work instead decouples metacognitive regulation from the policy itself, implementing regulation through decision corrections and belief feedback incorporated into the interaction context. This allows the same regulator to be integrated with different policy models.

Q1

To evaluate cross-environment generalization, we retained only training data originating from ProcTHOR and trained a new regulator using the same training procedure. We evaluated it on the AI2-THOR subset of SpatialWorld, with the results shown below.

ModelDailyWorkEntertainTravelOverall
Qwen3.5-27B5.5%9.8%5.0%0.0%5.8%
+ Spatial-JARVIS (ProcTHOR only)23.7%34.1%20.0%27.3%24.8%
Qwen3.5-Plus18.7%36.6%20.0%18.2%21.2%
+ Spatial-JARVIS (ProcTHOR only)23.3%41.5%27.5%27.3%26.4%

Spatial-JARVIS trained exclusively on ProcTHOR data provides overall improvements for both policy models on the AI2-THOR subset. These results indicate that the learned monitoring and regulation capabilities transfer from ProcTHOR environments to AI2-THOR scenes, supporting the cross-environment generalization of our method.

We sincerely thank you again for your thoughtful review. We hope our clarifications and additional experiments have addressed your concerns. If any questions remain, we would greatly appreciate your feedback and would be happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.