- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes H3D-LVR, which interleaves scene-layout, object-level, and view-level latent states with textual reasoning in multi-view VLMs. During SFT, VGGT-derived teacher features condition latent-pad inputs and supervise reconstruction. Inference uses self-generated states without VGGT or external feature inputs. Replay-aware, span-masked GRPO further improves answer accuracy.
Critical strengths
- The empirical gains are substantial. H3D-LVR-SFT improves MindCube-Tiny accuracy from 29.3% to 71.9% and SPAR-Bench from 38.8% to 72.9% over Qwen2.5-VL-7B, with further gains after RL. These improvements are obtained without external 3D features at inference.
- The component, data-quality, RL, and inference-time ablations provide meaningful evidence. Suppressing individual generated spans from the same checkpoint reduces accuracy, supporting their inference-time contribution. The object-reconstruction comparison also supports improved instance-specific feature alignment.
Critical weaknesses
- The controls do not fully separate hierarchy from additional supervision and teacher-feature conditioning. A text-only CoT SFT baseline using the same examples and reasoning annotations, and a single-level latent baseline with comparable supervision and capacity, would better isolate the hierarchical design. Leave-one-component-out variants retain multiple levels, and the external 3DThinker comparison is not a matched training control. Teacher features are both pad inputs and reconstruction targets during SFT; the feature-free evaluation supports useful generated states, but does not fully explain how this conditioning transfers to independent state construction.
- Semantic preservation during RL is supported indirectly. Table 3 shows answer-accuracy benefits from replay and masking, but does not directly measure latent semantics before and after RL. Section 4.6 describes object-slot averaging and sample-local cosine retrieval; clarification of whether these states are generated without teacher inputs is important for interpreting that evidence.
Overall, the substantial gains and meaningful ablations support the integrated method sufficiently for a weak accept, close to the acceptance boundary. Matched controls and direct semantic measurements would strengthen the interpretation, but I do not regard the training-inference difference alone as an established correctness flaw.
Questions for authors
- Could the authors provide matched text-only CoT SFT and single-level latent results, controlling examples, reasoning annotations, optimization budget, and latent capacity where applicable?
- Please clarify whether retrieval uses feature-free generated states, the evaluation/candidate selection, and the chance baseline. Could the same retrieval or reconstruction measures be reported before and after RL, including replay with and without masking? Measurements on feature-free generated states would also help assess the training-inference gap.
5 comments
W1: Teacher features are both pad inputs and reconstruction targets during SFT; the feature-free evaluation supports useful generated states, but does not fully explain how this conditioning transfers to independent state construction. & Q2: Measurements on feature-free generated states would also help assess the training-inference gap.
We appreciate this concern. We do not input any VGGT/teacher features at inference; the model generates the latent spans and their hidden states from the image and question alone. During SFT, VGGT features serve both as inputs to the latent span and as reconstruction targets; the hidden state immediately before each pad is trained to encode the corresponding spatial information. At inference, the first pad in a span receives the hidden state from the preceding position, and each subsequent pad directly receives the hidden state produced by the preceding pad. The continuous state is never decoded into text and re-encoded, and VGGT features are not injected during this process. During training, feature-reconstruction supervision requires the model to form a hidden state from the available context that can predict the VGGT feature; at inference, the model uses the hidden state from the preceding position to continue generating subsequent pads, so it can generate latent states under the same rule without teacher features.
The complete free-generation result in Figure 3 provides direct evidence: the model receives only the image and question, autonomously generates the reasoning text, latent spans, and answer, and we then extract the object states from the generated trajectory. The complete model obtains an Object Top-1 of 75.26%, whereas removing object feature-reconstruction supervision gives 39.76%. This shows that, without teacher features, the model can autonomously generate object states containing information about the current entity and associate the entity mentioned in the text with the corresponding instance in the same scene.
W2: Semantic preservation during RL is supported indirectly. Table 3 shows answer-accuracy benefits from replay and masking, but does not directly measure latent semantics before and after RL. Section 4.6 describes object-slot averaging and sample-local cosine retrieval; clarification of whether these states are generated without teacher inputs is important for interpreting that evidence. & Q2: Please clarify whether retrieval uses feature-free generated states, the evaluation/candidate selection, and the chance baseline. Could the same retrieval or reconstruction measures be reported before and after RL, including replay with and without masking?
Thank you for this detailed suggestion. Object retrieval does not receive any teacher/VGGT features; for each checkpoint, the model receives only the image and question and autonomously generates the full reasoning text, latent spans, and answer. We extract the object states from the generated trajectory and use object features extracted by VGGT for retrieval scoring.
All configurations in Table M use the same free-generation protocol and frozen object projection head; we compare SFT, Answer-only RL, Replay-no-mask, Replay+mask (1e-5), and Replay+mask (5e-6) in terms of answer accuracy and Object Top-1. For retrieval, the two slots of each object span are mapped through the original projection head and averaged to form an object query (the object representation to be retrieved). We collect valid VGGT object features from the same sample as candidates and ensure that the candidate set contains the annotated object. We compute the cosine similarity between the query and each candidate feature and select the candidate with the highest similarity; the query is counted as a Top-1 hit if that candidate is the annotated object.
The text-only and flat latent comparisons address my main concern
Thank you for the additional experiments and clarifications. The matched text-only CoT baseline and the comparison with a flat latent representation address my main concern about what the hierarchical design actually contributes. My original rating of 3 already reflects my positive assessment of the paper, and I will keep it as is.
We sincerely appreciate your recognition of our work! We are deeply grateful for the time and thoughtful effort you have dedicated to offering such detailed and constructive feedback. Your valuable suggestions have significantly enhanced both the scholarly rigor and presentation of our manuscript. We now address each point in detail below.
Thank you for this important comment. In Table K, we train Text-CoT-SFT and Text-CoT-RL without latent spans, using the same backbone, Gemini CoT training data, and SFT/RL budgets; this also corresponds to simultaneously removing the layout, object, and view latent branches. Text-CoT-SFT raises the overall MindCube accuracy from 29.3% to 61.8%, whereas H3D-LVR-SFT in the paper reaches 71.9%, a gain of 10.1 percentage points; on SPAR, Text-CoT-SFT obtains 61.4% and H3D-LVR-SFT obtains 72.9%, a gain of 11.5 points. These results show that textual CoT helps organize the reasoning steps, while the layout, object, and view states provide spatial evidence for the global scene, the current entity, and the observation viewpoint that text alone does not provide.
After RL, Text-CoT-RL obtains 68.3%/67.9% on MindCube/SPAR, whereas H3D-LVR-RL reaches 75.7%/77.4%, corresponding to gains of 7.4/9.5 points. Hierarchical generation and replay allow answer optimization to use the layout, object, and view states generated by the model, while full-span masking concentrates the direct policy/KL updates on the textual reasoning and answer positions that use these states. The gap is preserved in both SFT and RL, showing that the hierarchical latent representations provide supervised spatial information and remain usable as reasoning context during answer optimization. This highlights H3D-LVR's ability to connect spatial representation learning with language-level decision making.
Table K: Text-CoT baselines
In Table L, we control the samples, reasoning annotations, optimization budget, and, where applicable, the number of latent pads requested by the reviewer. We place the layout, object, and view pads in one flat span before the CoT. The single-level model obtains 65.3% on MindCube, compared with 71.9% for the complete hierarchy; on rotation/among/around, the two settings obtain 65.0%/60.3%/72.8% and 76.5%/67.3%/79.2%, respectively. The 6.6-point difference in overall accuracy shows that generating one global spatial state at once is insufficient when later reasoning must refer to different entities and viewpoints. H3D-LVR generates object/view states at the corresponding reasoning positions, keeping each continuous state aligned with the entity and viewpoint involved in the current judgment and thereby providing finer-grained spatial evidence.
Table L: Single-level 3D latent comparison