acceptodds
Under review as a conference paper at ICLR 2027

H3D-LVR: Hierarchical 3D Latent Visual Reasoning for Multi-View VLMs

Abstract

Vision-language models have achieved strong progress in visual perception and instruction following, yet multi-view 3D spatial reasoning remains challenging. Although textual chain-of-thought can expose intermediate reasoning steps, it operates at the output level and does not directly constrain the model to form spatially correct intermediate hidden states. Existing 3D or latent visual reasoning methods provide spatial supervision, but typically rely on a single-level latent representation, which is insufficient to capture the multi-granular information required for multi-view reasoning. We propose H3D-LVR, a hierarchical 3D latent visual reasoning framework that organizes a VLM's reasoning trajectory around three generated latent states: scene layout, key objects, and input views. During SFT, VGGT-derived layout features and matched object/view features supervise the corresponding spans through reconstruction losses. The corresponding features also fill latent-pad inputs under teacher forcing, but no such features are provided at inference. We further adapt answer-level GRPO to this hierarchical latent format with hidden-state replay and full-span loss masking, allowing reinforcement learning to improve answer behavior without corrupting the learned latent-token semantics. During inference, H3D-LVR uses only the input images and prompt; it does not run VGGT, receive precomputed layout/object/view features or annotations, or invoke external 3D tools. Experiments on MindCube-Tiny, SPAR-Bench, and MMSI-Bench demonstrate that H3D-LVR achieves substantial performance gains, strongly validating its effectiveness.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Multimodal Models, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oghnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes H3D-LVR, which interleaves scene-layout, object-level, and view-level latent states with textual reasoning in multi-view VLMs. During SFT, VGGT-derived teacher features condition latent-pad inputs and supervise reconstruction. Inference uses self-generated states without VGGT or external feature inputs. Replay-aware, span-masked GRPO further improves answer accuracy.

Critical strengths
  1. The empirical gains are substantial. H3D-LVR-SFT improves MindCube-Tiny accuracy from 29.3% to 71.9% and SPAR-Bench from 38.8% to 72.9% over Qwen2.5-VL-7B, with further gains after RL. These improvements are obtained without external 3D features at inference.
  2. The component, data-quality, RL, and inference-time ablations provide meaningful evidence. Suppressing individual generated spans from the same checkpoint reduces accuracy, supporting their inference-time contribution. The object-reconstruction comparison also supports improved instance-specific feature alignment.
Critical weaknesses
  1. The controls do not fully separate hierarchy from additional supervision and teacher-feature conditioning. A text-only CoT SFT baseline using the same examples and reasoning annotations, and a single-level latent baseline with comparable supervision and capacity, would better isolate the hierarchical design. Leave-one-component-out variants retain multiple levels, and the external 3DThinker comparison is not a matched training control. Teacher features are both pad inputs and reconstruction targets during SFT; the feature-free evaluation supports useful generated states, but does not fully explain how this conditioning transfers to independent state construction.
  2. Semantic preservation during RL is supported indirectly. Table 3 shows answer-accuracy benefits from replay and masking, but does not directly measure latent semantics before and after RL. Section 4.6 describes object-slot averaging and sample-local cosine retrieval; clarification of whether these states are generated without teacher inputs is important for interpreting that evidence.

Overall, the substantial gains and meaningful ablations support the integrated method sufficiently for a weak accept, close to the acceptance boundary. Matched controls and direct semantic measurements would strengthen the interpretation, but I do not regard the training-inference difference alone as an established correctness flaw.

Questions for authors
  1. Could the authors provide matched text-only CoT SFT and single-level latent results, controlling examples, reasoning annotations, optimization budget, and latent capacity where applicable?
  2. Please clarify whether retrieval uses feature-free generated states, the evaluation/candidate selection, and the chance baseline. Could the same retrieval or reconstruction measures be reported before and after RL, including replay with and without masking? Measurements on feature-free generated states would also help assess the training-inference gap.
5 comments

We sincerely appreciate your recognition of our work! We are deeply grateful for the time and thoughtful effort you have dedicated to offering such detailed and constructive feedback. Your valuable suggestions have significantly enhanced both the scholarly rigor and presentation of our manuscript. We now address each point in detail below.

W1: The controls do not fully separate hierarchy from additional supervision and teacher-feature conditioning. A text-only CoT SFT baseline using the same examples and reasoning annotations, and a single-level latent baseline with comparable supervision and capacity, would better isolate the hierarchical design. Leave-one-component-out variants retain multiple levels, and the external 3DThinker comparison is not a matched training control. & Q1: Could the authors provide matched text-only CoT SFT and single-level latent results, controlling examples, reasoning annotations, optimization budget, and latent capacity where applicable?

Thank you for this important comment. In Table K, we train Text-CoT-SFT and Text-CoT-RL without latent spans, using the same backbone, Gemini CoT training data, and SFT/RL budgets; this also corresponds to simultaneously removing the layout, object, and view latent branches. Text-CoT-SFT raises the overall MindCube accuracy from 29.3% to 61.8%, whereas H3D-LVR-SFT in the paper reaches 71.9%, a gain of 10.1 percentage points; on SPAR, Text-CoT-SFT obtains 61.4% and H3D-LVR-SFT obtains 72.9%, a gain of 11.5 points. These results show that textual CoT helps organize the reasoning steps, while the layout, object, and view states provide spatial evidence for the global scene, the current entity, and the observation viewpoint that text alone does not provide.

After RL, Text-CoT-RL obtains 68.3%/67.9% on MindCube/SPAR, whereas H3D-LVR-RL reaches 75.7%/77.4%, corresponding to gains of 7.4/9.5 points. Hierarchical generation and replay allow answer optimization to use the layout, object, and view states generated by the model, while full-span masking concentrates the direct policy/KL updates on the textual reasoning and answer positions that use these states. The gap is preserved in both SFT and RL, showing that the hierarchical latent representations provide supervised spatial information and remain usable as reasoning context during answer optimization. This highlights H3D-LVR's ability to connect spatial representation learning with language-level decision making.

Table K: Text-CoT baselines

MethodrotationamongaroundMindCube allObjRel OC-MVObjRel OO-MVSPAR allMMSI
Qwen2.5-VL-7B38.829.521.429.344.832.138.825.9
Text-CoT-SFT56.558.269.861.863.559.061.426.7
H3D-LVR-SFT76.567.379.271.975.869.872.927.7
Text-CoT-RL65.066.273.068.370.567.367.927.6
H3D-LVR-RL76.073.082.075.779.874.877.429.1

In Table L, we control the samples, reasoning annotations, optimization budget, and, where applicable, the number of latent pads requested by the reviewer. We place the layout, object, and view pads in one flat span before the CoT. The single-level model obtains 65.3% on MindCube, compared with 71.9% for the complete hierarchy; on rotation/among/around, the two settings obtain 65.0%/60.3%/72.8% and 76.5%/67.3%/79.2%, respectively. The 6.6-point difference in overall accuracy shows that generating one global spatial state at once is insufficient when later reasoning must refer to different entities and viewpoints. H3D-LVR generates object/view states at the corresponding reasoning positions, keeping each continuous state aligned with the entity and viewpoint involved in the current judgment and thereby providing finer-grained spatial evidence.

Table L: Single-level 3D latent comparison

rotationamongaroundMindCube all
Text-CoT-SFT56.558.269.861.8
Flat-multi-target latent-SFT65.060.372.865.3
H3D-LVR-SFT76.567.379.271.9

W1: Teacher features are both pad inputs and reconstruction targets during SFT; the feature-free evaluation supports useful generated states, but does not fully explain how this conditioning transfers to independent state construction. & Q2: Measurements on feature-free generated states would also help assess the training-inference gap.

We appreciate this concern. We do not input any VGGT/teacher features at inference; the model generates the latent spans and their hidden states from the image and question alone. During SFT, VGGT features serve both as inputs to the latent span and as reconstruction targets; the hidden state immediately before each pad is trained to encode the corresponding spatial information. At inference, the first pad in a span receives the hidden state from the preceding position, and each subsequent pad directly receives the hidden state produced by the preceding pad. The continuous state is never decoded into text and re-encoded, and VGGT features are not injected during this process. During training, feature-reconstruction supervision requires the model to form a hidden state from the available context that can predict the VGGT feature; at inference, the model uses the hidden state from the preceding position to continue generating subsequent pads, so it can generate latent states under the same rule without teacher features.

The complete free-generation result in Figure 3 provides direct evidence: the model receives only the image and question, autonomously generates the reasoning text, latent spans, and answer, and we then extract the object states from the generated trajectory. The complete model obtains an Object Top-1 of 75.26%, whereas removing object feature-reconstruction supervision gives 39.76%. This shows that, without teacher features, the model can autonomously generate object states containing information about the current entity and associate the entity mentioned in the text with the corresponding instance in the same scene.

W2: Semantic preservation during RL is supported indirectly. Table 3 shows answer-accuracy benefits from replay and masking, but does not directly measure latent semantics before and after RL. Section 4.6 describes object-slot averaging and sample-local cosine retrieval; clarification of whether these states are generated without teacher inputs is important for interpreting that evidence. & Q2: Please clarify whether retrieval uses feature-free generated states, the evaluation/candidate selection, and the chance baseline. Could the same retrieval or reconstruction measures be reported before and after RL, including replay with and without masking?

Thank you for this detailed suggestion. Object retrieval does not receive any teacher/VGGT features; for each checkpoint, the model receives only the image and question and autonomously generates the full reasoning text, latent spans, and answer. We extract the object states from the generated trajectory and use object features extracted by VGGT for retrieval scoring.

All configurations in Table M use the same free-generation protocol and frozen object projection head; we compare SFT, Answer-only RL, Replay-no-mask, Replay+mask (1e-5), and Replay+mask (5e-6) in terms of answer accuracy and Object Top-1. For retrieval, the two slots of each object span are mapped through the original projection head and averaged to form an object query (the object representation to be retrieved). We collect valid VGGT object features from the same sample as candidates and ensure that the candidate set contains the annotated object. We compute the cosine similarity between the query and each candidate feature and select the candidate with the highest similarity; the query is counted as a Top-1 hit if that candidate is the annotated object.

Reviewer_oghnOpenReview ↗

The text-only and flat latent comparisons address my main concern

Thank you for the additional experiments and clarifications. The matched text-only CoT baseline and the comparison with a flat latent representation address my main concern about what the hierarchical design actually contributes. My original rating of 3 already reflects my positive assessment of the paper, and I will keep it as is.

Reviewer_mDFmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes H3D-LVR, a hierarchical 3D latent visual reasoning framework for multi-view spatial reasoning. It introduces three types of latent states to represent scene layouts, key objects, and camera views. The method uses VGGT features to supervise these latent states during SFT. It further adopts replay-aware GRPO with latent-span masking to improve reasoning performance. At inference, the model requires no external 3D tools or additional geometric features. Experiments on three benchmarks show significant improvements over the baseline model.

Critical strengths
  1. The paper introduces three types of latent states for scene layouts, key objects, and camera views. These states are interleaved with textual reasoning and supervised by different 3D features. This design extends existing single-level latent reasoning methods and allows the model to capture spatial information at different granularities.
  2. The authors construct a feature-aligned training pipeline to connect textual reasoning with 3D spatial representations. They use object localization and cross-view feature aggregation to obtain object-level supervision. They also introduce crop-aware filtering to remove unreliable object features. This improves the quality of spatial supervision.
  3. The proposed reinforcement learning strategy combines hidden-state replay with full-span loss masking. Hidden-state replay maintains consistent latent contexts during policy evaluation. Full-span masking prevents answer-level rewards from directly affecting latent token semantics. This provides a practical solution for applying GRPO to hierarchical latent reasoning.
Critical weaknesses
  1. The authors use Gemini-2.5-Pro to generate structured CoT data and VGGT to provide spatial supervision. However, they do not compare H3D-LVR with a baseline trained on the same CoT data without latent tokens or feature reconstruction. The reported improvements may therefore partly come from the additional reasoning supervision rather than the hierarchical latent representations. This comparison is necessary to establish the actual contribution of the proposed method.
  2. The authors claim that hierarchical latent representations capture more fine-grained spatial information than existing single-level representations. However, the experiments only remove individual latent components. They do not compare the proposed hierarchy with a single-level 3D latent representation under the same training conditions. It remains unclear whether the hierarchical design itself provides advantages over a simpler latent reasoning approach.
  3. H3D-LVR-RL achieves only 29.1% accuracy on MMSI-Bench, compared with 25.9% for the original Qwen2.5-VL-7B model. This improvement is much smaller than those on MindCube-Tiny and SPAR-Bench. The model also performs below several existing methods on MMSI-Bench. These results raise concerns about its generalization to more diverse spatial reasoning tasks. Further analysis is needed to explain the inconsistent improvements across benchmarks.
Questions for authors

See weakness.

4 comments

We sincerely thank you for the time and careful consideration you have given to providing detailed and constructive feedback. Your valuable insights have greatly improved both the technical accuracy and clarity of our manuscript. Below, we respond to each of your comments point by point.

W1: The authors use Gemini-2.5-Pro to generate structured CoT and VGGT for spatial supervision, but provide no baseline trained on the same CoT data without latent tokens or feature reconstruction. The reported improvements may therefore partly come from the additional reasoning supervision rather than the hierarchical latent representations.

Thank you for this valuable suggestion. We train Text-CoT-SFT and Text-CoT-RL without latent spans, using the same backbone, Gemini CoT training data, and SFT/RL budgets; this also corresponds to simultaneously removing the layout, object, and view latent branches. Text-CoT-SFT raises the overall MindCube accuracy from 29.3% to 61.8%, whereas H3D-LVR-SFT in the paper reaches 71.9%, a gain of 10.1 percentage points; on SPAR, Text-CoT-SFT obtains 61.4% and H3D-LVR-SFT obtains 72.9%, a gain of 11.5 points. These results show that textual CoT helps organize the reasoning steps, while the layout, object, and view states provide spatial evidence for the global scene, the current entity, and the observation viewpoint that text alone does not provide.

After RL, Text-CoT-RL obtains 68.3%/67.9% on MindCube/SPAR, whereas H3D-LVR-RL reaches 75.7%/77.4%, corresponding to gains of 7.4/9.5 points. Hierarchical generation and replay allow answer optimization to use the layout, object, and view states generated by the model, while full-span masking concentrates the direct policy/KL updates on the textual reasoning and answer positions that use these states. The gap is preserved in both SFT and RL, showing that the hierarchical latent representations provide supervised spatial information and remain usable as reasoning context during answer optimization. This highlights H3D-LVR's ability to connect spatial representation learning with language-level decision making.

Table H: Text-CoT baselines

MethodrotationamongaroundMindCube allObjRel OC-MVObjRel OO-MVSPAR allMMSI
Qwen2.5-VL-7B38.829.521.429.344.832.138.825.9
Text-CoT-SFT56.558.269.861.863.559.061.426.7
H3D-LVR-SFT76.567.379.271.975.869.872.927.7
Text-CoT-RL65.066.273.068.370.567.367.927.6
H3D-LVR-RL76.073.082.075.779.874.877.429.1

W2: The authors claim that hierarchical latent representations capture finer-grained spatial information than existing single-level representations, but provide no direct comparison with a single-level 3D latent representation under the same training conditions. It remains unclear whether the hierarchical design itself provides advantages over a simpler latent reasoning approach.

Thank you for this helpful suggestion. We place the layout, object, and view latent pads in one flat span before the CoT. The single-level model obtains 65.3% on MindCube, compared with 71.9% for the complete hierarchy; on rotation/among/around, the two settings obtain 65.0%/60.3%/72.8% and 76.5%/67.3%/79.2%, respectively. The 6.6-point difference in overall accuracy shows that generating one global spatial state at once is insufficient when later reasoning must refer to different entities and viewpoints. H3D-LVR generates object/view states at the corresponding reasoning positions, keeping each continuous state aligned with the entity and viewpoint involved in the current reasoning and thereby providing finer-grained spatial evidence.

Table I: Single-level 3D latent comparison

rotationamongaroundMindCube all
Text-CoT-SFT56.558.269.861.8
Flat-multi-target latent-SFT65.060.372.865.3
H3D-LVR-SFT76.567.379.271.9

W3: H3D-LVR-RL achieves only 29.1% accuracy on MMSI-Bench, compared with 25.9% for the original Qwen2.5-VL-7B model. This improvement is much smaller than those on MindCube-Tiny and SPAR-Bench. The model also performs below several existing methods on MMSI-Bench. These results raise concerns about its generalization to more diverse spatial reasoning tasks. Further analysis is needed to explain the inconsistent improvements across benchmarks.

We appreciate this question. First, the InternVL2.5-8B result of 28.7% and the H3D-LVR-SFT result of 27.7% are not a matched comparison; the training gain of H3D-LVR should be measured on the same Qwen2.5-VL-7B backbone, from 25.9% to 27.7% to 29.1%. MMSI consists of 1,000 multiple-choice questions drawn from more than 120,000 images. Different subsets evaluate multi-image entity correspondence, scene reconstruction, attribute judgment, camera/object motion judgment, and spatial-logic reasoning. Together, these tasks cover multiple spatial-reasoning steps from entity localization to scene-relation judgment, which makes the benchmark a demanding compositional set of spatial capabilities. GPT-5 obtains about 40% in the official evaluation [1], showing that even a powerful general-purpose model has difficulty completing these steps reliably; scores from different models therefore occupy a relatively narrow range.

The table below covers 3B, 7B, 32B, and 72B models: H3D-LVR uses about 14K training examples, while Spatial-MLLM[2] uses about 120K and RoboBrain2.0[3] about 5.3M; the MMSI-Bench results are 27.7% for Spatial-MLLM-3B and 28.5% for RoboBrain2.0-32B, while Qwen2.5-VL-72B reaches only 30.7%. Despite the large differences in model scale and training-data size, the results remain within 25.9%–30.7%, indicating that the benchmark’s overall scores are jointly constrained by its compositional spatial-reasoning difficulty. Within this range, H3D-LVR improves the same Qwen2.5-VL-7B from 25.9% to 29.1% after SFT and RL, a gain of 3.2 points. This shows that H3D-LVR’s layout/object/view hierarchy and its corresponding RL optimization can apply the multi-view spatial representations formed during training to out-of-domain MMSI reasoning, yielding strong cross-benchmark generalization.

Table G: MMSI-Bench results

Qwen2.5-VL-7BH3D-LVR-SFTH3D-LVR-RLSpatial-MLLM-3BRoboBrain2.0-32BQwen2.5-VL-72B
Model size7B7B7B3B32B72B
MMSI-Bench25.927.729.127.728.530.7

[1] MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence.

[2] Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence.

[3] RoboBrain 2.0 Technical Report.

My concerns have been well solved, I will raise my score to 3. Good luck!

We thank the reviewer for the positive assessment and are pleased that our response has clarified your concerns.

Reviewer_pDScOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes H3D-LVR, which fine-tunes Qwen2.5-VL-7B to emit three kinds of latent spans inside its chain of thought: one layout span (12 pads) at the start, an object span (2 pads) after each key-object mention, and a view span (1 pad) after each image reference. During SFT, VGGT-derived features fill the pad inputs under teacher forcing and also serve as reconstruction targets. The layout target is dense VGGT tokens, objects use box-pooled features from Kimi-K2.5 boxes with crop-aware filtering, and views use per-image features. The SFT data comes from MindCube and SPAR training sets, with structured CoT generated by Gemini-2.5-Pro. Answer-level GRPO follows, using hidden-state replay and loss masking over the spans. Inference uses only images and the prompt. Reported results: MindCube-Tiny 29.3 → 75.7, SPAR 38.8 → 77.4, MMSI 25.9 → 29.1.

Critical strengths
  1. Sensible, deployable design. The layout, object, and view latent slots are tied to explicit text anchors. VGGT is used only during training, so inference needs no 3D tools or extra inputs.
  2. Large in-domain gains. MindCube-Tiny rises from 29.3 to 75.7 and SPAR from 38.8 to 77.4, beating Qwen2.5-VL-32B and 3DThinker.
  3. Careful engineering with targeted ablations. The crop-aware and box-size filtering, the replay-plus-masking RL, and the object-retrieval probe each get an ablation. The probe shows that reconstruction makes object states informative (75.3 vs. 39.8 top-1).
Critical weaknesses
  1. The key baseline is missing. H3D-LVR is fine-tuned on MindCube and SPAR training data, while most baselines are zero-shot. No baseline trains the same backbone on the same Gemini CoT data without latent spans, so the contribution of the latents cannot be separated from in-domain SFT. The MindCube paper itself reports large gains from plain SFT and RL on its training set. On the out-of-domain MMSI benchmark, the gain is small (+1.8 after SFT), and the SFT model (27.7) trails InternVL2.5-8B (28.7).
  2. The causal role of the latent content is not shown. The component ablations remove one branch at a time; no variant removes all three. The inference ablation blocks span generation by setting the span-start logit to −∞-\infty, which pushes the model off its training distribution, so the resulting drops say little about what the latents encode. There is no control that replaces pad states with zeros, random vectors, or another sample's states.
  3. The train–inference gap is unclear. During SFT, the pads are filled with projected ground-truth VGGT features, so the text that follows is conditioned on true 3D information. At inference, the model must fill the pads itself, but the paper does not say what is fed in at those positions or measure the gap. It is also unclear whether the retrieval probe (Fig. 3) uses free generation.
  4. Reporting is thin and claims are overstated. Ablations cover only MindCube and appear to be single runs, with 1–3 point differences on 1,050 questions. The size of the SFT corpus is not reported. Part of the RL gain comes from a learning-rate change. Relative to 3DThinker, LVR, and CoVT, the novelty is mainly the three-level split, so the claim to be the "first to decouple" is overstated.
Questions for authors
  1. How does a text-only baseline perform: same backbone, same Gemini CoT data, same SFT and RL budget, no latent spans? Please also report the SFT corpus size and any scene overlap with the MindCube-Tiny and SPAR-Bench test sets.
  2. What exactly fills the pad positions at inference? How does accuracy change if the pad states are replaced with zeros, random vectors, or another sample's states? Is the retrieval probe run under free generation?
  3. How much do the ablations in Tables 2–5 vary across seeds? Given greedy decoding, what differs across the "three runs" averaged in Table 1?
5 comments

We sincerely appreciate the time and effort you have devoted to providing detailed and constructive feedback. Your insightful comments have been invaluable in improving both the technical quality and clarity of our manuscript. Below, we address each of your points individually.

W1: H3D-LVR is fine-tuned on MindCube and SPAR training data, whereas most baselines use zero-shot settings; no baseline trains the same backbone on the same Gemini CoT data while removing latent spans, so the latent contribution cannot be separated from in-domain SFT. & W2: Each component ablation removes only one branch; there is no variant that removes all three branches simultaneously. & Q1: What is the performance of a text-only reasoning baseline trained with the same backbone, Gemini CoT data, and SFT/RL budgets, but without latent spans?

Thank you for this valuable suggestion. We train Text-CoT-SFT and Text-CoT-RL without latent spans, using the same backbone, Gemini CoT training data, and SFT/RL budgets; this also corresponds to simultaneously removing the layout, object, and view latent branches. Text-CoT-SFT raises the overall MindCube accuracy from 29.3% to 61.8%, whereas H3D-LVR-SFT in the paper reaches 71.9%, a gain of 10.1 percentage points; on SPAR, Text-CoT-SFT obtains 61.4% and H3D-LVR-SFT obtains 72.9%, a gain of 11.5 points. These results show that textual CoT helps organize the reasoning steps, while the layout, object, and view states provide spatial evidence for the global scene, the current entity, and the observation viewpoint that text alone does not provide.

After RL, Text-CoT-RL obtains 68.3%/67.9% on MindCube/SPAR, whereas H3D-LVR-RL reaches 75.7%/77.4%, corresponding to gains of 7.4/9.5 points. Hierarchical generation and replay allow answer optimization to use the layout, object, and view states generated by the model, while full-span masking concentrates the direct policy/KL updates on the textual reasoning and answer positions that use these states. The gap is preserved in both SFT and RL, showing that the hierarchical latent representations provide supervised spatial information and remain usable as reasoning context during answer optimization. This highlights H3D-LVR's ability to connect spatial representation learning with language-level decision making.

Table A: Text-CoT baselines

MethodrotationamongaroundMindCube allObjRel OC-MVObjRel OO-MVSPAR allMMSI
Qwen2.5-VL-7B38.829.521.429.344.832.138.825.9
Text-CoT-SFT56.558.269.861.863.559.061.426.7
H3D-LVR-SFT76.567.379.271.975.869.872.927.7
Text-CoT-RL65.066.273.068.370.567.367.927.6
H3D-LVR-RL76.073.082.075.779.874.877.429.1

W2: The inference ablation sets the span-start logit to negative infinity to prevent span generation, moving the model off the training distribution. There is no control that replaces pad states with zeros, random vectors, or another sample's states. & Q2: How does accuracy change after replacing pad states with zeros, random vectors, or states from another sample?

We appreciate this helpful suggestion. Using the Full rollout in Table 5 of main paper as the baseline, we preserve the layout-span boundaries and pad count while replacing the layout states with zero vectors or norm-matched random vectors. Full rollout obtains 71.90% on MindCube, No layout obtains 64.95%, layout-zero replacement obtains 66.75%, and layout-random replacement obtains 65.41%. The zero and random replacements remain 5.15 and 6.49 percentage points below Full rollout, respectively, showing that the gain comes from layout states generated by the model and matched to the current scene; these states provide a global spatial reference for subsequent relation reasoning.

In addition, in the experiment reported in Figure 3 of the main paper, the complete model obtains a hit rate of 75.26%, while removing object feature-reconstruction supervision gives 39.76%. This provides complementary evidence that the hidden states of the object pads retain information about the current scene entity and can distinguish which object is being discussed among the candidate objects in the same scene. Together, these experiments show that layout and object states participate in subsequent spatial reasoning rather than serving as formal continuous placeholders.

Table B: Layout-state replacement controls

Inference settingrotationamongaroundMindCube all
No layout64.0062.1771.2064.95
Full rollout + layout zero65.5062.8373.2566.75
Full rollout + layout norm-matched random62.5062.3371.5065.41
Full rollout76.5067.3379.2071.90

W3: During SFT, pads are filled with projected real VGGT features; at inference, the model must fill them autonomously, but the actual pad inputs and the gap between training and inference are not specified; it is also unclear whether Figure 3's retrieval probe uses free generation. & Q2: What is filled at pad positions at inference, and does the retrieval probe use free generation?

Thank you for raising this point. At inference, the actual input at a pad position is the hidden state from the preceding position, i.e., xt=h(t−1)x_t = h_{(t−1)}. Once the model enters a latent span, the decoder outputs only the prescribed number of pads: 12 for layout, 2 for object, and 1 for view. The pad ID marks a position for continuous computation; the information being propagated is the hidden state produced at each step. The hidden state is not first decoded into text and then re-encoded as the next context. Instead, the continuous state is passed directly from one pad to the next, and readable reasoning text is generated after leaving the span.

Consistent with this inference procedure, SFT feature reconstruction reads the hidden state immediately before each pad: the first slot of an object span uses the state at obj_start to predict the object feature, and the second slot uses the state at the first pad to continue the prediction. Thus, the training target constrains the same states that are propagated during inference. The retrieval probe in Figure 3 also uses complete free generation: the model receives only the image and question, autonomously generates the reasoning text, latent spans, and answer, and we then read the object states from the generated trajectory. VGGT object features are extracted afterward as candidate targets, and the generated object states are mapped through the original projection head for retrieval scoring.

W4: The SFT corpus size is not reported. & Q1: Please report the SFT corpus size and whether it overlaps at the scene level with the MindCube-Tiny and SPAR-Bench test sets?

Thank you for this clarification. MindCube and SPAR use 10,000 and 4,645 training examples, respectively. During training-data extraction, scenes belonging to the MindCube-Tiny and SPAR-Bench test sets were excluded, so there is no scene overlap between the training and test sets.

W4: The ablations cover only MindCube and appear to be single-run; on 1,050 questions, some differences are only 1–3 points. & Q3: How much do the ablations in Tables 2–5 vary across seeds? Since greedy decoding is used, what differs across the three runs averaged in Table 1?

We appreciate this concern. We did conduct three independent training runs with different seeds and greedy evaluation, but due to an oversight when filling the original manuscript, the reported values corresponded to one run. We have now added the complete results from the three training seeds and report mean ± std in the tables below. To address the concern that the ablations cover only MindCube, Table C additionally reports multi-seed ablations on SPAR. We report representative results in the text: for the component ablation, Full obtains 72.92% ± 1.70 on MindCube and 73.06% ± 1.61 on SPAR; for the RL ablation, Replay+mask (1e-5) obtains 75.39% ± 0.87, and reducing the learning rate to 5e-6 gives 76.44% ± 1.41. The ordering of the means and the overall trends of these settings agree with the single-run results, showing that the ablation conclusions are stable across training seeds. The single-run ablations in the main paper provide the effect comparisons, while the added mean ± std results quantify training variation.

Table C: SFT component ablation studies (avg3)

rotationamongaroundMindCube allObjRel OC-MVObjRel OO-MVSPAR all
w/o Layout70.83 ± 3.0663.22 ± 1.5873.25 ± 4.1367.83 ± 1.5671.08 ± 1.7762.23 ± 1.7666.89 ± 0.60
w/o Object63.67 ± 3.6266.50 ± 1.3273.08 ± 2.9668.22 ± 0.7572.50 ± 1.6461.96 ± 0.4267.50 ± 0.99
w/o View62.17 ± 2.9365.61 ± 1.2377.50 ± 3.4769.00 ± 1.2474.33 ± 2.0264.64 ± 1.0569.73 ± 1.18
Full76.33 ± 2.2567.00 ± 1.7480.08 ± 1.6372.92 ± 1.7076.58 ± 1.5369.16 ± 1.7673.06 ± 1.61

Table D: RL ablation (avg3; same SFT initialization)

LRrotationamongaroundMindCube all
Answer-only1e-579.83 ± 4.1668.78 ± 1.0277.33 ± 2.9873.47 ± 0.79
Replay-no-mask1e-579.17 ± 5.8070.11 ± 1.5576.00 ± 2.1773.58 ± 1.53
Replay+mask1e-577.33 ± 2.0271.50 ± 2.1880.25 ± 1.1575.39 ± 0.87
Replay+mask5e-677.67 ± 4.6572.44 ± 1.6781.83 ± 2.5076.44 ± 1.41

Table E: BBox-size filtering ablation studies (avg3)

rotationamongaroundMindCube all
No filter73.17 ± 3.8260.39 ± 2.2078.42 ± 1.6368.53 ± 2.16
BBox3074.50 ± 5.2267.00 ± 1.5977.25 ± 2.2271.67 ± 1.02
BBox7067.50 ± 4.0966.28 ± 1.3577.92 ± 2.5069.36 ± 1.42
BBox5076.33 ± 2.2567.00 ± 1.7480.08 ± 1.6372.92 ± 1.70

Table F: Inference-time span ablations (avg3)

rotationamongaroundMindCube all
No layout63.33 ± 2.0262.89 ± 0.8271.17 ± 1.9465.72 ± 1.24
No object69.33 ± 2.7563.67 ± 1.4578.42 ± 3.0970.53 ± 1.11
No view65.17 ± 1.0467.44 ± 1.9378.83 ± 2.1070.86 ± 1.74
Full rollout76.33 ± 2.2567.00 ± 1.7480.08 ± 1.6372.92 ± 1.70

W4: Part of the RL gain comes from changing the learning rate.

Thank you for this important comment. Following the four settings in Table 3 of the main paper, we progressively add RL components while keeping the SFT initialization, prompt set, and answer/format rewards fixed. The SFT model obtains 71.90% on MindCube, while Answer-only RL obtains 72.76%, an improvement of 0.86 percentage points. This setting uses only the final-answer and format feedback to train textual decision making, showing that answer rewards can directly improve textual reasoning and decisions but do not directly teach the model how to use the continuous layout, object, and view states.

Adding hierarchical generation of these states and using hidden-state replay to reconstruct the same continuous trajectory during policy scoring raises accuracy to 73.52%, a further gain of 0.76 percentage points, showing that answer optimization now takes place on the spatial states actually generated by the model. At the same learning rate of 1e-5, adding full-span masking raises accuracy to 74.86%, 1.34 percentage points above Replay-no-mask; masking concentrates the update on the textual reasoning and answer positions that use these states, reducing interference from updates to the discrete structural markers. Thus, at LR=1e-5, Replay+mask improves over Answer-only by 2.10 percentage points. Reducing the learning rate of the complete configuration to 5e-6 gives 75.71%, a further gain of 0.85 percentage points.

In this comparison, the gains from replay and masking designed for the hierarchical continuous reasoning path are larger than the gain from lowering the learning rate alone, indicating that H3D-LVR's main benefit comes from adapting answer optimization to its spatial representation and reasoning path, while the smaller learning rate provides additional control over the update magnitude.

W4: Relative to 3DThinker, LVR, and CoVT, the novelty is mainly the three-level split, so the claim of “first to decouple” is overstated.

We appreciate this suggestion. LVR [1] generates question-relevant continuous states in a visual-embedding space and interleaves them with textual reasoning; 3DThinker [2] aligns 3D latents with VGGT features to form a 3D representation from multi-view inputs; CoVT [3] organizes continuous tokens through visual tasks such as segmentation, depth, edges, and DINO features. Our focus is different: for multi-view 3D scenes, the global scene provides a relatively stable spatial reference, while the reasoning process switches among entities and uses the observation viewpoint to determine spatial relations. We therefore organize layout, object, and view as hierarchical representations with separate targets, fixed generation positions, and bindings to the corresponding textual references. The model first forms a cross-view scene reference, then generates an entity-specific state when an entity is mentioned and a viewpoint-specific state when a viewpoint is referenced. The role of the hierarchy is not simply to increase the number of latent states; it keeps each continuous state aligned with the relation judgment it serves, providing more precise spatial evidence when the reasoning process switches entities or viewpoints. We will replace the phrase “first to decouple” and frame the contribution as organizing layout/object/view states for multi-view spatial relations, with each state generated at the corresponding reasoning step and made directly available to subsequent textual reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.