acceptodds
Under review as a conference paper at ICLR 2027

Thought–State Co-Evolution: Closed-Loop Cognitive Mapping for Spatial Reasoning

Abstract

Scene-level spatial reasoning requires multimodal large language models to integrate partial video observations into task-sufficient spatial states. Existing cognitive-map pipelines support this integration but often lack reasoning-to-state feedback: externally supplied entity inventories leave relevance assessment outside the reasoning process, while fixed state estimates prevent evidence-driven revision when inference reveals uncertainty. To close this loop, we propose Reasoning-in-the-Loop Cognitive Mapping (RiCM), a framework in which reasoning directs state expansion and verification, and updated states guide subsequent inference. A dedicated spatial tool uses instance-consistent 3D anchors to ground entity association and evidence-gated local revision, decoupling geometric estimation from high-level reasoning. Final-answer rewards alone, however, do not explicitly supervise the sufficiency of intermediate states. We therefore introduce Progressive State Sufficiency Training (PSST), which promotes early and sustained state sufficiency through evaluation by an independent reader and unbiased estimation of the fraction of a fixed interaction horizon covered by sufficient states. Its reward combines joint correctness of the policy and the terminal-state reader with an interaction penalty, enabling the policy to learn when to expand, verify, and stop without ground-truth maps or oracle entity inventories. Experiments on ReVSI and MMSI-Video-Bench show consistent gains, supporting the value of bidirectional reasoning-state feedback for spatial reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Spatial Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oPNPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how to improve global spatial reasoning by dynamically constructing and refining cognitive maps during reasoning. To address this problem, the authors propose RiCM, a closed-loop framework that enables reasoning-guided state expansion and verification. The authors further introduce PSST, a reinforcement learning method that encourages the model to construct sufficient spatial states with fewer interactions. Experiments on ReVSI and MMSI-Video-Bench demonstrate the effectiveness of the proposed method.

Critical strengths
  1. This paper focuses on an important research problem: how to construct and refine spatial memory during reasoning, rather than relying on pre-constructed static memories.
  2. The proposed framework is relatively complete. It integrates reasoning-guided state construction, IGGT prior, and reinforcement learning into a closed-loop spatial reasoning framework.
  3. The experiments are relatively comprehensive. The authors evaluate the proposed method on 2 benchmarks and conduct ablation studies to investigate the contributions of different components.
Critical weaknesses
  1. The written needs to be improved. Figure 1 cannot straightforwardly highlight the gap between local spatial awareness and global spatial reasoning. Figure 2 and 5 are also not clear and full of texts. Section 2 hindered the fluency of reading. The paper introduces too many technical terms and concepts, making it difficult to follow, such as reasoning-to-state feedback, state support, etc.
  2. The training process is not fully described. How to perform reject sampling? Does this paper performs cold-start training? If yes, how to construct the cold-start data?
  3. In Table 1 and 2, RiCM-8B performs sub-optimal in some subsets among non-proprietary methods. However, this paper does not fully analyze the reasons.
  4. The paper could add some related reference [1][2]. These papers also propose to actively perceiving environments and refine memory. The authors could discuss the difference with them.
  5. The related work section is placed in the appendix rather than the main text, hence the main text lacks a key section. The authors should also provide a more comprehensive discussion of the core difference in related work to better highlight the novelty of the proposed method.

[1] Thinking in 360: Humanoid visual search in the wild, CVPR 2026. [2] Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models, ICML 2026.

Questions for authors

Please see weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer oPNP (1/2)

Initial Acknowledgment

We are grateful to the reviewer for the valuable and encouraging feedback. The insightful suggestions and comments helped us identify areas for clarification and further improvement. In the following, we address each of the reviewer's concerns point by point.


Q1. Clarity of the figures, motivation, and terminology.

We address three aspects of the reviewer’s comment:

  • How Figure 1 highlights the gap between local spatial perception and global spatial reasoning.
  • Why Figures 2 and 5 use text labels to explain the mapping failures and the RiCM workflow.
  • Why Section 2 introduces the motivation and terminology before presenting the method.

1. Figure 1 — Making the human–model gap visible

Red and blue task labels distinguish global spatial reasoning from local spatial perception. The purple bars provide a common human reference across both groups, allowing readers to see where the gap widens. We will add a red outline to these bars to make the comparison easier to see.

2. Figures 2 and 5 — Explaining failures and component roles

The labels serve different purposes. Figure 2 identifies where each mapping failure arises. Figure 5 shows what each component contributes and how its output guides the next step. Appendices F and G connect these labels to specific prompt instructions and steps in worked examples.

3. Section 2 — Motivating the method before introducing it

We present the two controlled comparisons before the method to explain why RiCM needs both map expansion and targeted revision. Consistent terminology lets readers trace each problem to its corresponding mechanism in Section 3. We have added explicit pointers to Appendices B, F, and G, which ground these terms in object inventories, prompt instructions, and worked examples, respectively.


Q2. Rejection sampling and cold-start training

We address two questions:

  • How did we select the 80K training examples through rejection sampling?
  • How does PSST train the policy without cold-start demonstrations?

1. Rejection sampling — Selecting questions with answer disagreement

For each candidate question, we sampled 16 reasoning trajectories from Qwen3-VL-8B-Instruct with the cognitive-map tool disabled. We discarded the question if all 16 final answers agreed and retained it otherwise. This yielded 80K training examples for PSST.

2. Cold-start training — Learning from state sufficiency

We applied PSST directly to Qwen3-VL-8B-Instruct without task-specific cold-start training. RiCM allows the policy to make an incorrect intermediate estimate, seek further evidence, and revise it. The ground-truth answer does not specify which error the policy will make or which observation will resolve it, so it cannot supply a ground-truth correction trajectory.

The absence of a ground-truth correction trajectory is precisely why we introduced PSST. Different policies may make and recover from different intermediate errors, yet their maps can be evaluated by the same criterion: does the map already contain enough information to answer the question? PSST turns this criterion into a training signal by asking a frozen reader to answer from the question and map alone at a sampled checkpoint and at the end of the trajectory. This lets the policy learn when to expand, verify, or stop without requiring a prescribed sequence of reasoning steps.

AuthorsOpenReview ↗

Response to Reviewer oPNP (2/2)

Q3. Why does RiCM-8B trail other models on some subtasks?

We address two aspects of this question:

  • Which capabilities is RiCM designed to improve?
  • How does that scope relate to the results in Tables 1 and 2?

1. Design objective — Global spatial reasoning

current models can exhibit local spatial awareness yet struggle to form coherent global spatial representations.

Section 1 observes that current models “can exhibit local spatial awareness yet struggle to form coherent global spatial representations.” We therefore set an explicit objective: “We therefore focus on improving the global spatial reasoning capabilities of MLLMs.” RiCM uses a cognitive map to associate objects across views and verify spatial relations. It introduces no component or training objective specifically for local perception, so we would not expect it to consistently surpass leading open or proprietary models on local estimation tasks.

2. Subtask results — Where cognitive mapping helps

Table 1 shows the clearest gains over our 8B backbone on relative direction (39.5 to 73.5) and route planning (40.5 to 64.2), both of which require integrating spatial relations across views. Absolute distance (52.3 to 49.3) and object size (69.0 to 64.1) decline, underscoring the limits of the approach on metric estimation. Table 2 also shows variation within relational reasoning: RiCM improves instance–instance reasoning from 25.0 to 37.1, while GeoThinker-8B reaches 40.2. Our claim is thus specific to improved cross-view reasoning, with the largest gains on direction and planning.


Q4. Why the two methods cannot be evaluated directly on VSI-Bench or ReVSI?

Both methods study active perception for spatial reasoning, and we have added them to Related Work. Neither can be evaluated directly on VSI-Bench or ReVSI in its published form.

1. Thinking in 360° — Interactive panoramic search

The method selects head rotations, observes the resulting views in a 360° panorama, and returns a direction pointing toward an object or a path. [1] VSI-Bench and ReVSI provide prerecorded videos and score answers to spatial questions. Running this method would therefore require adapting both how it obtains observations and how it produces answers.

2. Active Exploring like a Pigeon — View retrieval with supplied spatial relations

The method uses textual descriptions of the transformations between viewpoints to select which view to retrieve. Neither VSI-Bench nor ReVSI supplies these descriptions. [2] Applying the method would therefore require estimating the transformations from the video or modifying the retrieval policy to operate without them.


Q5. Placement of Related Work and clarification of novelty.

The nine-page limit on the initial submission led us to place Related Work in Appendix A. With the additional page available during rebuttal, we will move Related Work into the main text and explicitly distinguish RiCM from the closest prior methods, including the two studies highlighted by the reviewer.

Section 2 (Motivation) already develops the rationale for RiCM through two controlled studies. The first examines how much task-relevant information a predefined object inventory supplies. The second compares independent map construction with reasoning-guided revision of uncertain estimates. Together, these studies motivate RiCM’s central contribution: during reasoning, the model determines both what is missing from the map and which existing estimates require verification.


References

[1] Heyang Yu et al. “Thinking in 360°: Humanoid Visual Search in the Wild.” CVPR, 2026, pp. 22445–22455. CVF Proceedings

[2] Wei Deng, Xianlin Zhang, and Mengshi Qi. “Active Exploring like a Pigeon: Reinforcing Spatial Reasoning via Agentic Vision-Language Models.” ICML, 2026, pp. 24407–24417. PMLR Proceedings


Final Acknowledgment

Thank you again for the time and care you put into reviewing our work. Your questions prompted us to revisit our results and explain our reasoning more clearly, and we appreciate how closely you read the paper. We hope our responses have addressed your concerns, and we wish you all the best in your research and future academic endeavors.

Reviewer_BoSvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RiCM, a closed-loop video spatial reasoning framework in which a policy dynamically queries a geometry-backed spatial tool to expand or verify its cognitive map during inference. To train the policy without ground-truth maps, the authors introduce Progressive State Sufficiency Training (PSST), which penalizes excessive tool calls while rewarding states that enable a frozen reader to answer the question correctly. The approach yields strong gains on ReVSI and MMSI-Video-Bench, particularly on relative direction and planning tasks

Critical strengths
  • The paper is well-organized, with detailed explanations of the preliminary, intuition, and methodology.
  • The core motivation of the paper is clear and compelling. Moving away from a static, one-shot cognitive map so that the reasoning process can actively spot missing details and request targeted corrections addresses a very real bottleneck in current spatial reasoning pipelines.
  • The paper provides comprehensive experiments that cleanly ablate expansion, revision, geometric anchoring, evidence gating, and reward design, alongside runtime analyses that clarify the computational overhead.
Critical weaknesses
  • PSST evaluate state sufficiency based on whether a frozen reader answers correctly. However, the reader can easily guess right or exploit question priors without relying on the map, or fail despite a correct state which means the reward tracks specific reader success rather than guaranteeing reliable spatial grounding.

  • The reward can be hard to encourage the useful progress in unsuccessful trajectories. Because both the intermediate and the main positive rewards depend on joint terminal success, intermediate improvements earn no positive credit whenever either the policy or the terminal reader fails. As a result, training may come to favor short failed trajectories over longer ones that build better states but still make an error at the end.

  • Several models used in the evaluation, including the GPT baselines, are outdated. Comparisons with more recent models would help assess whether the reported gains remain competitive.

  • As mentioned in the appendix, the qualitative examples are scripted instead of actual model behavior. These examples explain the intended workflow but can't demonstrate that the trained policy actually discovers missing entities, resolves ambiguity, or corrects its own map.

  • Some results are inconsistent, and uncertainty estimates are missing. Table 1 and Figures 7 and 8 report different scores for relative distance/direction and absolute distance. These discrepancies need clarification, while repeated runs or confidence intervals would help assess small gains such as 38.5 vs. 38.0 on MMSI-Video-Bench.

Questions for authors
  • How often does the frozen reader answer correctly with an empty, incomplete, or deliberately incorrect map?
  • How does RiCM compare with a baseline given the same retrieved frames and geometric information but no iterative feedback?
5 comments
AuthorsOpenReview ↗

Response to Reviewer BoSv (1/3)

Initial Acknowledgment

We are grateful to the reviewer for the valuable and encouraging feedback. The insightful suggestions and comments helped us identify areas for clarification and further improvement. In the following, we address each of the reviewer's concerns point by point.


W1 & Q1. Can the frozen Reader answer from priors, or from an empty, incomplete, or incorrect map?

Thank you for this question. The Reader is deliberately designed to test task sufficiency, i.e., whether the explicit state contains the information needed to answer the question. Several design choices make this test reliable.

1. Questions carry no exploitable scene prior, so the Reader refuses to answer without a map.

ReVSI questions specify what to determine but not the scene itself. Knowing what a desk is does not reveal how many desks a given room contains (e.g., ID 32: "How many desks are in the scene?"). Knowing what a toaster is does not reveal which side of the oven it occupies (e.g., ID 5545: "If I stand by the oven facing the TV, is the toaster to my front-left, front-right, back-left, or back-right?"). When given the question alone, with neither the map nor the video, the Reader responded that it had insufficient information in 97.8% of cases. A refusal yields v=0v=0 and hence no positive reward (Eq. 5). Any residual question-only guess is also identical across all rollouts of the same question, so it cancels in GRPO's group-normalized advantage. The only signal that survives normalization is the one that depends on the state.

2. No ground-truth cognitive map exists, so "incomplete" or "incorrect" maps are not well-defined.

Calling a map incomplete or incorrect presupposes a reference map that defines what is missing or wrong, and no such reference exists. The state Bi=(Si,Mi,Ui)\mathcal B_i=(\mathcal S_i,\mathcal M_i,\mathcal U_i) is constructed by the policy for each question (Sec. 3.1). The policy selects which entities and relations to include, adds auxiliary entities for association or disambiguation, and keeps competing hypotheses unresolved. Many different states can be equally sufficient. The training data therefore provide a ground-truth answer but no ground-truth cognitive map at any step. This is precisely why PSST evaluates what a state enables rather than how closely it matches a reference. 3D box annotations give object locations, but turning them into a "complete" map would require deciding externally which entities are relevant. That is exactly the oracle state specification (Sec. 2) that RiCM is designed to eliminate. A hand-crafted corrupted map would therefore measure the experimenter's own choices, not the Reader's behavior on policy-generated states.

3. Joint success makes the reward conservative by design.

Positive reward requires both the policy and the terminal-state Reader to answer correctly from the same final state. The Reader cannot see the video, the interaction history, or the policy's answer, so a correct Reader answer must be supported by the state itself. Because the policy must also be correct, a state that merely looks sufficient earns no reward. Every positive reward is thus tied to a state that actually supports correct answering. PSST achieves this without ground-truth maps or oracle inventories.

AuthorsOpenReview ↗

Response to Reviewer BoSv (2/3)

W2. Does the reward fail to credit useful progress in unsuccessful trajectories?

Thank you for this question. Gating positive reward on joint terminal success is a deliberate design choice in PSST.

1. Progress in a failed trajectory cannot be verified, so it should not be rewarded.

Without a ground-truth cognitive map (see W1), the only evidence that a state is good is that it supports correct answering. A failed trajectory falls into one of two cases. If the terminal Reader fails (vK=0v_K=0), the final state is insufficient, and nothing shows that it is "better" than a shorter one. If the Reader succeeds but the policy fails, the policy did not use the state correctly, and rewarding the state would decouple state construction from reasoning. As stated in Sec. 4.3, "transient sufficiency cannot compensate for an inadequate final state." Partial credit would instead invite reward hacking through inflated intermediate updates. The bonus λvˉJ\lambda\bar v_J is designed to rank successful trajectories by how early and how persistently they reach sufficiency.

2. Short failures are never preferred over longer successes.

With λ=1\lambda=1, β=0.2\beta=0.2, and Imax⁡=7I_{\max}=7:

  • Every successful trajectory receives R≥0.8R\ge 0.8.
  • Every failed trajectory receives R∈[−0.2,0]R\in[-0.2,0].

The success–failure gap is therefore at least four times the entire spread among failures, and it dominates GRPO's group-normalized advantage. Each additional update costs only β/Imax⁡≈0.029\beta/I_{\max}\approx 0.029. Any update that raises the success probability by more than about 2.9 points therefore increases the expected reward. The penalty removes only interaction that does not help.

3. Empirically, the policy does not collapse to short trajectories.

  • Interaction scales with task demand. The policy averages 3.79 and 3.68 tool calls on route planning and relative direction, versus 0.00 on counting and object size.
  • More interaction keeps helping. In Fig. 9, accuracy increases with the interaction budget up to seven updates.
  • The penalty removes only waste. Raising β\beta from 0.2 to 0.4 cuts updates by 35.7% while changing the Overall score by only 0.2 points (App. E.3).

W3. Comparison with more recent models

Thank you for the suggestion. We evaluated GPT-5.4 on ReVSI under the same protocol as Table 1 (64 frames):

MethodObj.CntA.DistObj.SizeR.SizeRel.DistRel.DirR.PlanOverall
GPT-5.470.340.273.359.857.843.533.354.0
RiCM-8B (Ours)55.447.364.154.662.673.264.260.2
Δ−14.9+7.1−9.2−5.2+4.8+29.7+30.9+6.2

Built on an 8B open-source backbone, RiCM outperforms GPT-5.4 by 6.2 points overall. The advantage comes from global spatial reasoning: RiCM leads by 29.7 points on relative direction and 30.9 points on route planning. Both tasks require cross-view integration and relational composition. GPT-5.4's advantages lie in local perception tasks such as counting and size estimation, consistent with the local–global gap shown in Fig. 1. We will add GPT-5.4 to Table 1 in the revised paper.


W4. Are the case studies actual model behavior?

Thank you for raising this point. The wording in our original appendix was ambiguous, and we are glad to clarify it. All three cases are actual inference outputs of the trained RiCM-8B on ReVSI. The raw outputs are long JSON records interleaved with verbose reasoning text, so we edited them for readability. We removed the verbose reasoning and reorganized the remaining content into a uniform step-by-step format: the policy's Expand/Verify request, the returned map state, the unresolved set, and the final answer at each step. This editing changes only the presentation. The actions, state updates, and outcomes are exactly those produced by the model. The term "scripted" referred to this reorganization for presentation, not to fabricated behavior. To remove any ambiguity, we have revised the appendix to state that the cases are condensed and reorganized from recorded runs.

AuthorsOpenReview ↗

Response to Reviewer BoSv (3/3)

W5. Inconsistent results and uncertainty estimates.

1. Why Table 1 differs from Figures 7 and 8.

Thank you for the careful check. Table 1 reports our main evaluation, while Figures 7 and 8 come from a re-evaluation of the full method run alongside the ablations. We deliberately used greedy decoding for the policy so that re-runs would reproduce identical scores. However, the cognitive-map generator had a separate decoding configuration with sampling enabled, an implementation detail omitted from the submission. Different map estimates can lead the policy to different follow-up requests or answers. The task-level pattern confirms this cause: tasks that do not invoke the generator are identical across runs, and differences appear only in generator-heavy tasks.

ReVSI taskMain eval.Re-eval. (Figs. 7–8)ΔMean generator calls
Obj. Count55.455.40.00.00
Abs. Distance49.347.3−2.02.21
Obj. Size64.164.10.00.00
Room Size54.654.60.00.14
Rel. Distance61.962.6+0.72.46
Rel. Direction73.573.2−0.33.68
Route Planning64.264.20.03.79
Overall60.460.2−0.21.79

We have revised Table 1 and the text to use the re-evaluation results consistently. This is the more conservative of the two runs, and no conclusion changes. A detailed trace is given in our response to Reviewer KUyH (1/3).

2. Uncertainty and the size of the gains.

The two independent runs above directly measure run-to-run variation. Both runs share the same model weights, but the map generator samples independently in each. Their Overall scores differ by only 0.2 points (60.4 vs. 60.2). RiCM improves on its Qwen3-VL-8B backbone by 11.1 points on ReVSI, about 50 times this variation. It also improves by 10.9 points on MMSI-Video-Bench. The largest task-level changes are equally robust: relative direction gains 33.7 points and route planning gains 23.7 points over the backbone.

Regarding 38.5 vs. 38.0 on MMSI-Video-Bench, we do not claim that RiCM outperforms Gemini 3 Pro. Gemini 3 Pro is a proprietary frontier model of undisclosed scale and training data, so a head-to-head margin is not a like-for-like comparison. What this result shows is that an 8B open-source model, trained with RiCM and PSST, performs on par with a leading proprietary model. The controlled comparison is against the shared backbone (+10.9) and the strongest open-source baseline, VLM-3R-7B (+4.2).


Q2. Comparison with a baseline that has the same geometric information but no iterative feedback.

This baseline is the static cognitive map variant in RQ2 (Fig. 7). It shares RiCM's spatial tool, geometric information, evidence access, and interaction limit. Its map is built once and kept fixed, so reasoning cannot expand or revise it.

Variant (ReVSI)Iterative feedbackOverall
Qwen3-VL-8B backbone–49.1
Static cognitive map–52.3
Expansion only✓55.9
Revision only✓55.0
RiCM (full feedback)✓60.2

Without feedback, the static map improves the backbone by only 3.2 points. Iterative feedback adds a further 7.9 points. The largest gains are on tasks that require relational composition: relative direction (+16.2), relative distance (+20.5), and route planning (+15.6). Sec. 2 (Fig. 4) gives consistent evidence under matched observation budgets. Without feedback, positional accuracy does not improve, while reasoning-guided refinement raises it from 30.2 to 55.4. The gains therefore come from the feedback loop, not from additional geometric information.


Final Acknowledgment

Thank you again for the time and care you put into reviewing our work. Your questions prompted us to revisit our results and explain our reasoning more clearly, and we appreciate how closely you read the paper. We hope our responses have addressed your concerns, and we wish you all the best in your research and future academic endeavors.

Reviewer_KUyHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces RiCM, a closed-loop framework that interleaves reasoning steps with dynamic, evidence-gated 3D state refinement to address the decoupling of perception and reasoning in multi-view spatial understanding. To train this framework without dense 3D ground-truth, the authors propose PSST, which employs a frozen text-only reader and uniform checkpoint sampling to evaluate and reward early state sufficiency. The proposed paradigm is evaluated across multiple 3D spatial benchmarks, demonstrating marked performance gains on relation-compositional tasks such as relative direction and route planning.

Critical strengths

Compelling and rigorous motivation: The motivational experiments in Figures 2, 3, and 4 are exceptionally well-designed. By strictly aligning computational budgets, they convincingly isolate and quantify both the artificial performance gains introduced by oracle entity inventories and the ineffectiveness of isolated, feedforward mapping for geometric precision, pinpointing the critical bottlenecks in current multimodal spatial mapping.

Sound formulation of intermediate sufficiency reward (PSST): Utilizing an independent, frozen reader isolated from video inputs as an operational judge of state sufficiency effectively eliminates policy shortcuts via visual memory. Furthermore, coupling this with uniform trajectory checkpoint sampling minimizes gradient variance while maintaining an elegant computational budget of at most two reader evaluations per rollout.

Critical weaknesses

Numerical inconsistencies and dubious data reliability: In Table 1, the reported average for RiCM across the 7 sub-tasks on ReVSI does not mathematically match the mean of its individual scores. Moreover, there is an unexplained discrepancy between the RiCM numbers reported in Table 1 and those plotted in Figure 8. These notable data inconsistencies cast severe doubts on the empirical reliability of the reported results.

Absence of direct comparisons with targeted prior method: The paper's core motivation and thesis are built upon critiquing the limitations of existing cognitive-mapping paradigms. However, none of these specific, criticized cognitive-mapping frameworks are included as empirical baselines in the main experiments. The lack of these essential baselines leaves the claimed superiority over the current state of the art insufficiently substantiated.

Questions for authors

Discrepancy in experimental data: In Table 1, the reported overall mean of RiCM does not align with the average of its 7 sub-categories on ReVSI, and the metrics in Table 1 contradict the data points shown in Figure 8. Could the authors explain why these critical numerical inconsistencies exist?

Degradation under dynamic environments: How does RiCM degrade when dynamic, moving objects are present in the scene, and what is its quantitative performance under such conditions?

Reader exploitation and metric validity: Is it possible that the policy generates geometrically incorrect coordinates, yet the text-only Reader guesses the correct answer by exploiting commonsense priors in the question context, thereby receiving an unearned positive reward? Have the authors statistically evaluated the false-positive rate where vi=1v_i = 1 despite severe metric errors in MiM_i?

5 comments
AuthorsOpenReview ↗

Response to Reviewer KUyH (1/3)

Initial Acknowledgment

We are grateful to the reviewer for the valuable and encouraging feedback. The insightful suggestions and comments helped us identify areas for clarification and further improvement. In the following, we address each of the reviewer's concerns point by point.


W1&Q1. Reconciling the ReVSI Results in Table 1 and Figure 8.

Nice catch, and thank you for checking both the arithmetic in Table 1 and the task-level scores in Figure 8. You are correct on both counts. The seven RiCM scores printed in Table 1 average to 60.4 when rounded to one decimal place, although the table reports 60.2. Three of those scores also differ from Figure 8. This puzzled us for some time because we had expected a second evaluation of the full method to reproduce the first exactly. After tracing both evaluations and the final proofreading step, we confirmed that 60.4 and 60.2 are both correct for their respective runs. The reporting error occurred when we combined results from the two runs in Table 1.


1. Why the two evaluations differed.

When preparing Figure 8, we evaluated the full RiCM configuration again alongside the ablated variants. As stated in RQ3, “All variants use the same policy and map-generator weights, initial video input, evidence access permissions, and interaction limit.” One implementation detail we omitted from the submission is that the reasoning policy used greedy decoding in both evaluations. With fixed inputs and deterministic inference throughout the pipeline, we would expect the same prediction on every question and therefore identical scores.

During our investigation, we found that the cognitive-map generator had a separate decoding configuration with sampling enabled. Even when the policy issued the same request, the generator could return a different estimate of an object’s position or a spatial relation. That estimate became part of the policy’s next input. Greedy decoding applied to these different map states could therefore produce different follow-up requests or final answers.

ReVSI taskmain evaluationrerun for Figure 8Score changeMean tool calls
Object Counting55.455.40.00.00
Absolute Distance49.347.3−2.02.21
Object Size64.164.10.00.00
Room Size54.654.60.00.14
Relative Distance61.962.6+0.72.46
Relative Direction73.573.2−0.33.68
Route Planning64.264.20.03.79
Overall60.460.2−0.21.79

The task-level pattern is consistent with variation in map generation. Object Counting and Object Size never invoked the map generator, and Room Size averaged only 0.14 calls per question. All three scores agreed across runs at the reported precision. Of the four tasks averaging more than two calls per question, three showed score changes. Route Planning scored 64.2 in both runs despite averaging 3.79 calls per question. Absolute Distance showed the largest change, at 2.0 points. It is the only numerical-answer task among the three affected tasks. A shift in estimated distance can cross a scoring threshold, whereas a shift in estimated geometry can leave the selected multiple-choice answer unchanged for Relative Distance or Relative Direction.


2. How the error entered Table 1.

The seven displayed scores from the original main evaluation average to 60.4286, so the original Table 1 Overall of 60.4 was correct. During final proofreading, we saw that this value differed from the 60.2 in Figure 8, mistook the difference for a transcription error, and changed that single cell while retaining the seven task scores from the original run.


We have revised Table 1 and the corresponding text throughout the paper to use the full-method results from Figure 8 consistently. The corrected entries are 47.3 for Absolute Distance, 62.6 for Relative Distance, and 73.2 for Relative Direction. The seven displayed scores now average to 60.2, matching the reported Overall. We have also corrected the relative-direction gain over the listed backbone from 34.0 to 33.7 points (73.2 − 39.5).

AuthorsOpenReview ↗

Response to Reviewer KUyH (2/3)

W2. Direct Comparison with Prior Cognitive-Mapping Methods.

We address this concern in two parts:

  • limitations of transferring oracle-dependent mapping protocols to new benchmarks.
  • comparison with prior cognitive-mapping methods on VSI-Bench and the possibility of benchmark-specific fitting.

1. Oracle-dependent protocols cannot be transferred to benchmarks that lack the required annotations.

As Section 2 explains under Oracle state specification:

Existing cognitive-map methods often assume task-relevant entities and relations are known before map construction, treating state support as an external input.

Appendix B (Annotation-Provided Object Inventories in Cognitive Mapping) documents a concrete example: MindCube’s Plain-CGMap-Out prompt reads the object categories from each sample’s annotations before constructing the map. On a benchmark that provides only the video and question, reproducing this protocol requires supplying the missing object lists. RiCM selects the entities to represent during reasoning and requires no such annotations at inference time.


2. RiCM leads both comparison methods on VSI-Bench.

The table compares RiCM with the results for Map2Thought [1] and Cog3DMap [2] across all eight VSI-Bench tasks.

MethodOverallObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppr. Order
Map2Thought61.070.855.070.169.456.969.838.157.4
Cog3DMap-8B65.169.654.867.867.164.885.643.067.9
RiCM-8B (ours)67.272.859.266.258.772.183.454.870.4

Appendix E.2 (VSI-Bench Performance and Cross-Benchmark Generalization) examines the possibility of benchmark-specific fitting. For example, Cambrian-S leads RiCM by 13.5 points on room-size estimation on VSI-Bench but trails it by 7.9 points on ReVSI. This reversal suggests that some gains on VSI-Bench may be specific to its evaluation protocol. On VSI-Bench itself, RiCM achieves the highest overall score among the three methods (67.2), ahead of Map2Thought by 6.2 points and Cog3DMap by 2.1 points. Its largest advantage is on route planning, where the respective margins are 16.7 and 11.8 points.


Q2. How does RiCM perform when objects move?

The cognitive-mapping task studied in VSI-Bench was designed for static scene understanding. Given a video of an indoor environment, the model places object centers on a single 10 × 10 grid and uses their positions to reason about distances. The camera viewpoint can change throughout the video, but the map represents a single scene layout. Tracking an object’s trajectory falls outside the scope of this representation.

RiCM’s mapping tool shares this static-layout assumption, and we trained the policy on static-scene data. RiCM invokes the tool adaptively. When a question asks how an object moved, the policy can answer directly from the sequence of video frames. To assess this behavior, we report accuracy and mean map-tool calls per question on the two MMSI-Video-Bench subsets that explicitly concern moving objects.

Dynamic subsetQuestionsQwen3-VL-8B-Instruct accuracyRiCM accuracyDifferenceMean map-tool calls
Instance Motion9022.2%22.2%0.0 points0.09
Interactive Motion8129.6%28.4%−1.2 points0.13

RiCM matched the backbone on Instance Motion and answered one fewer question correctly on Interactive Motion. The policy rarely invoked the static map on either subset, averaging 0.09 and 0.13 calls per question, respectively. On these motion questions, RiCM’s accuracy remained close to that of its backbone.


References

[1] Xiangjun Gao, Zhensong Zhang, Dave Zhenyu Chen, Songcen Xu, Long Quan, Eduardo Pérez-Pellitero, and Youngkyoon Jang. “Map2Thought: Explicit 3D Spatial Reasoning via Metric Cognitive Maps.” CVPR Findings, pp. 7154–7164, 2026.

[2] Chanyoung Gwak, Yoonwoo Jeong, Byungwoo Jeon, Hyunseok Lee, Jinwoo Shin, and Minsu Cho. “Cog3DMap: Multi-View Vision-Language Reasoning with 3D Cognitive Maps.” arXiv:2603.23023, 2026.

AuthorsOpenReview ↗

Response to Reviewer KUyH (3/3)

Q3. Can the Reader reward a geometrically incorrect map by guessing from priors?

This is possible in principle. We address two questions:

  • Why we cannot directly measure the requested false-positive rate?
  • Whether the Reader can reliably answer from commonsense priors alone?

1. Why we cannot directly measure the requested false-positive rate?

The requested rate is the fraction of states with severe geometric errors for which the Reader nevertheless gives the correct answer. Measuring it requires first identifying those erroneous states independently of the Reader. As described in Section 4.1, our training data provide a ground-truth answer for each question, but no ground-truth cognitive map at each reasoning step. The policy selects and revises entities and relations as reasoning proceeds, so the state to be evaluated also changes from step to step. Without independent geometric labels for these states, we cannot directly compute the requested rate across training trajectories.

This missing supervision is precisely what motivated PSST. Instead of requiring a reference map at every step, PSST uses the supervision already available: the ground-truth answer. We hand the intermediate state and question to a frozen Reader that has no access to the video, interaction history, or policy answer, and test whether it can recover the correct answer. This makes task sufficiency trainable throughout the trajectory. A positive trajectory reward further requires both the policy and the terminal-state Reader to answer correctly.


2. Can the Reader answer these questions from commonsense priors alone?

ReVSI questions specify what to determine while providing almost no information about the particular scene. The same wording can have different correct answers in different scenes: knowing what a desk is cannot tell the Reader how many desks are in this room, and knowing what a toaster is cannot tell it which side of the oven the toaster occupies. Commonsense priors alone therefore cannot reliably determine the correct answer. The examples below illustrate this across all seven tasks in the 64-frame split, with wording shortened for readability.

TaskExample question
Object counting (ID 32)How many desks are in the scene?
Absolute distance (ID 2541)What is the distance in meters between the closest points of the TV and the radiator?
Object size (ID 1134)What is the longest dimension of the TV, in centimeters?
Room size (ID 6680)What is the area of the scene in square meters, combining the rooms if more than one is shown?
Relative distance (ID 4261)Which object is farthest from the floor lamp: the TV, microwave, standing fan, or oven?
Relative direction (ID 5545)If I stand by the oven facing the TV, is the toaster to my front-left, front-right, back-left, or back-right?
Route planning (ID 6217)Starting at the room entrance facing the dresser, which way should I turn before walking straight to the table: left, right, or back?

We tested the frozen Reader with only the question, withholding both the cognitive map and the video. In 97.8% of cases, it responded that it had insufficient information to answer. This result indicates that the Reader could not reliably answer these scene-specific questions from commonsense priors alone.


Final Acknowledgment

Thank you again for the time and care you put into reviewing our work. Your questions prompted us to revisit our results and explain our reasoning more clearly, and we appreciate how closely you read the paper. We hope our responses have addressed your concerns, and we wish you all the best in your research and future academic endeavors.

Comments on the paper

3 comments

Dear Reviewers,

The authors have posted their responses to your comments and questions. Please review their replies and share any follow-up comments or questions, especially if any concerns remain unresolved or need further clarification.

Your engagement is important to ensuring a constructive and productive discussion. Thank you for your time and contributions to the ICLR review process.

Best regards,
The AC

AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair (1/2)

We sincerely thank all reviewers (BoSv, KUyH, oPNP) and the Area Chair for their time and constructive feedback.


1. Reviewers' Recognition of Our Contributions

All three reviewers recognized the novelty and significance of the core idea: spatial reasoning should actively construct and refine its own cognitive map, rather than consume a static, pre-built one.

Novel and important problem formulation.

  • BoSv: "The core motivation of the paper is clear and compelling. Moving away from a static, one-shot cognitive map so that the reasoning process can actively spot missing details and request targeted corrections addresses a very real bottleneck in current spatial reasoning pipelines."
  • oPNP: "This paper focuses on an important research problem: how to construct and refine spatial memory during reasoning, rather than relying on pre-constructed static memories."

Rigorous motivation that pinpoints the bottleneck.

  • KUyH: "The motivational experiments in Figures 2, 3, and 4 are exceptionally well-designed. By strictly aligning computational budgets, they convincingly isolate and quantify both the artificial performance gains introduced by oracle entity inventories and the ineffectiveness of isolated, feedforward mapping for geometric precision, pinpointing the critical bottlenecks in current multimodal spatial mapping."

Sound and efficient training method (PSST).

  • KUyH: "Utilizing an independent, frozen reader isolated from video inputs as an operational judge of state sufficiency effectively eliminates policy shortcuts via visual memory ... while maintaining an elegant computational budget of at most two reader evaluations per rollout."
  • oPNP: "The proposed framework is relatively complete. It integrates reasoning-guided state construction, IGGT prior, and reinforcement learning into a closed-loop spatial reasoning framework."

Comprehensive experiments with strong gains.

  • BoSv: "The paper provides comprehensive experiments that cleanly ablate expansion, revision, geometric anchoring, evidence gating, and reward design, alongside runtime analyses that clarify the computational overhead." The approach "yields strong gains on ReVSI and MMSI-Video-Bench, particularly on relative direction and planning tasks."
  • KUyH: "marked performance gains on relation-compositional tasks such as relative direction and route planning."
  • oPNP: "The experiments are relatively comprehensive."

Clear organization.

  • BoSv: "The paper is well-organized, with detailed explanations of the preliminary, intuition, and methodology."
AuthorsOpenReview ↗

General Response to All Reviewers and the Area Chair (2/2)

2. Key Concerns Resolved During the Rebuttal

None of the concerns questions the core idea. The three most substantive concerns, and how each was resolved with new evidence, are summarized below.

ConcernRaised byResolution
Reliability of the reported numbersKUyH, BoSvRoot cause identified; Table 1 corrected; no conclusion changes; run variation 0.2 vs. gain +11.1
Validity of the PSST rewardKUyH, BoSvQuestion-only Reader refuses in 97.8% of cases; joint-success gating; reward-gap analysis
Baselines and source of the gainsKUyH, BoSvOutperforms GPT-5.4, Map2Thought, and Cog3DMap; static-map ablation isolates the feedback loop

2.1 Reliability of the results (KUyH, BoSv)

Concern: Table 1 and Fig. 8 reported slightly different scores, and the Overall in Table 1 did not match the mean of its task scores.

Resolution: We identified the exact root cause. The policy uses greedy decoding, but the cognitive-map generator had sampling enabled, so the re-evaluation for Fig. 8 differed slightly from the main run. A proofreading step then mixed the two runs in Table 1. The task-level evidence confirms this cause precisely: tasks that never invoke the generator (Object Counting, Object Size) are identical across runs, and differences appear only on generator-dependent tasks. Table 1 now uses a single run throughout, and its task scores average exactly to the reported Overall.

Outcome: No conclusion changes. The two runs also provide a direct measure of run-to-run variation: 0.2 points, about 50× smaller than RiCM's gain over its backbone (+11.1 on ReVSI, +10.9 on MMSI-Video-Bench). The gains on the targeted relational tasks are larger still (+33.7 on relative direction, +23.7 on route planning).

2.2 Validity of the PSST reward (KUyH, BoSv)

Concern: The frozen Reader might answer correctly from question priors, rewarding states that do not truly support the answer.

Resolution: We tested the Reader directly. Given only the question, with neither the map nor the video, it declines to answer in 97.8% of cases, because ReVSI questions are scene-specific (e.g., "How many desks are in the scene?"). Any residual question-only guess is identical across all rollouts of the same question and cancels in GRPO's group-normalized advantage, so only state-dependent signal drives learning. Positive reward further requires both the policy and the Reader to answer correctly from the same final state.

Outcome: The reward cannot be earned from priors. Our reward-gap analysis also shows that failed trajectories are never preferred over successful ones: the success–failure gap is at least 4× the entire spread among failures. Empirically, the policy uses more tool calls exactly where tasks demand it (3.8 calls on route planning vs. 0.0 on counting).

2.3 Baselines and the source of the gains (KUyH, BoSv)

Concern: Comparisons with recent models and with prior cognitive-mapping methods were missing, and it was unclear whether the gains come from feedback or from extra geometric information.

Resolution and outcome:

  • Recent frontier model. RiCM-8B outperforms GPT-5.4 on ReVSI by 6.2 points overall, with +29.7 on relative direction and +30.9 on route planning.
  • Cognitive-mapping methods. On VSI-Bench, RiCM-8B (67.2) surpasses Map2Thought (61.0) and Cog3DMap-8B (65.1), with route-planning margins of +16.7 and +11.8.
  • Feedback, not extra information. A static-map variant with the same tool, geometry, evidence access, and interaction limit reaches only 52.3, versus 60.2 for RiCM. Iterative feedback alone contributes +7.9 points, which directly validates the paper's central thesis.

3. Summary

The reviewers agree that the paper tackles an important problem with a well-motivated, novel approach, supported by rigorous motivation studies, a sound training method, and comprehensive experiments. Every substantive concern has been resolved with concrete new evidence. The results are consistent and robust, the reward cannot be exploited through priors, RiCM outperforms GPT-5.4 and prior cognitive-mapping methods, and the feedback loop is confirmed as the source of the gains. Each of these results further strengthens the paper's central claims.

We are grateful to all reviewers for their careful reading, and we hope the AC will take the full discussion into account.

Sincerely,
The Authors of Submission 29417

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.