- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SGT-3D trains an RGB-only vision-language model to generate object-level 3D attributes and reason from those attributes. The method contains two main components. CS-OPSD distills supervision using continuations conditioned on the student’s own committed intermediate attributes, addressing the mismatch between ideal reference states during training and imperfect generated states during inference. GF-RL introduces geometry-aware rewards with field-specific normalization and temporal credit assignment.
The paper reports 69.2% accuracy on 3DSRBench-real and 96.2% on CV-Bench-3D. It also provides ablations against answer-only reinforcement learning, attribute cross-entropy, masking variants, different training orders, and alternative reward-credit schemes. The method and results are promising, but the statistical reliability, leakage analysis, and independent annotation protocol are not documented sufficiently for the strength of the claims.
Critical strengths
-
CS-OPSD addresses a meaningful training-inference mismatch. Supervising continuations from the student’s own committed attributes is better aligned with deployment than always conditioning on correct reference attributes.
-
The ablation design is unusually comprehensive. Comparisons against answer-only RL, attribute CE, teacher-conditioning variants, alternate training orders, and crossed reward controls make the reported improvement easier to attribute to the proposed components.
-
The independent geometry evaluation and joint geometry-answer metrics provide evidence beyond final-answer accuracy. This is important for verifying whether the model actually uses the proposed 3D representation.
Critical weaknesses
-
The method depends heavily on privileged 3D labels, regenerated traces, and a frozen expert during training. This dependency limits scalability and makes the quality and provenance of those labels central to the method, but the paper does not discuss it in sufficient depth.
-
The proposed object representation may be too narrow for some of the reasoning claims. Center, size, and orientation do not explicitly represent gravity, visibility, occlusion, or geometric uncertainty. The paper notes that camera-up is not necessarily world-up but does not explain how world-relative height questions are resolved.
-
Several comparisons combine differences in objectives, supervision, training history, and inference procedure. They show that the complete SGT-3D pipeline performs well, but do not always isolate which methodological component is responsible.
Questions for authors
- How are gravity-relative questions answered when camera-up does not correspond to world-up?
- What information about occlusion and geometric uncertainty is preserved in the object representation?
- Which baselines use the same initial checkpoint and supervision resources as SGT-3D?
3 comments
Response to Reviewer PGhx [Part 2/2]
W3. Isolating the methodological component responsible for each gain
Our ablations isolate each methodological component: every variant starts from the same checkpoint, uses the same supervision resources and changes one component at a time. Variants A–G and every ablation row start from Qwen2.5-VL-7B-Instruct. They use SpatialReasoner's training data, the same regenerated references and matched update budgets (App. A.5). The committed-state teacher is +4.9 QA over plain conditioning and +5.1 over expert priority (Table 5). The GF-RL objective is +3.8 QA over answer-only RL and +2.6 over answer RL with attribute cross-entropy (Table 3). Boundary credit is +3.0 QA over segment credit (Table 6).
Q3. Baselines with the same initial checkpoint and supervision resources as SGT-3D
All models in Table 2 use the same Qwen2.5-VL-7B backbone as SGT-3D, and SpatialReasoner is the only external baseline that also shares our training-data source. The revised Table 2 includes only Qwen2.5-VL-7B-based models, so that the backbone is held fixed across all comparisons. The other baselines use different supervision resources: VST-7B uses VST-P (4.1M samples) and VST-R (135K samples), 3DThinker-7B uses 237K SFT and 57K RL examples, and VERDICT adds no training and uses frozen verifiers at inference. PolyV and Mini-o3 originally use other settings, so we align them with the dense, RGB-only, tool-free inference setting of Table 2: PolyV-dense is our reproduction on the Qwen2.5-VL-7B backbone without MoE experts, and Mini-o3 keeps its entropy-regularized training but disables tools at inference (App. C.1).
With the same backbone and the same data source, SGT-3D exceeds SpatialReasoner by 8.9 points (69.2 vs. 60.3). It also exceeds 3DThinker-7B (66.3) and VST-7B-RL (60.0), although it uses only 25,200 spatial training records, far fewer than either.
Follow-up on Rebuttal
Dear Reviewer PGhx,
Thanks again for your thoughtful feedback.
We would greatly appreciate any further discussion of our rebuttal responses. We would be happy to provide further clarification and address remaining concerns.
Thank you so much for your time!
Best regards,
Authors
Response to Reviewer PGhx [Part 1/2]
We thank the reviewer for the assessment that "CS-OPSD addresses a meaningful training-inference mismatch" and that the ablation design is "unusually comprehensive".
W1. Scalability, replaceability and label provenance of the training-time expert
The dependency on the 3D expert is computationally cheap, so it does not limit scalability, and SGT-3D does not require the expert labels to be exact.
Dependency and scalability. On 8$\times$H200 GPUs, the frozen 3D expert labels one image in 0.19 s. Labeling the whole training set and generating all reasoning traces takes 79 minutes, about 1% of the total training time. The method uses the expert only through the reference ZR (centers, dimensions, orientations), so any model that outputs these fields can replace it.
Provenance and quality of the labels. The labels come from one automatic pipeline, and SGT-3D remains accurate when they are imperfect.
W2 / Q1 / Q2. Gravity, occlusion and geometric uncertainty in the object representation
The proposed object representation is general, and it handles both cases that the reviewer describes.
Gravity (Q1). Gravity-relative questions are answered along an estimated world-up, not along the camera axis. Think estimates world-up uc from scene cues, such as upright people, vertical edges and support surfaces, and compares Δh=uc⊤(cA−cB) (App. A.1, Eq. 5). The camera translation cancels in Δh, so only the up direction is needed. In Fig. 6, where the image is rotated by 180°, this gives the correct order (+0.550), whereas camera-up gives the opposite order (−0.538). In Fig. 7, an unrotated photo, SGT-3D correctly answers that a cat on a bench is higher than a boat on the water, although the cat appears lower in the image. SGT-3D also has the best Height accuracy in Table 2 (60.6), +8.1 over SpatialReasoner.
Visibility, occlusion and geometric uncertainty (W2, Q2). The representation preserves the hidden extent of each object: Gaze predicts a full-object (amodal) 3D box, so its center and dimensions include occluded parts. The hidden height moves the box center downward along gravity (Eq. 6), and Think compares distances between these complete centers. In Fig. 8, a couch hides the lower stem and base of a floor lamp, and the question asks which lamp is closer to the couch. SGT-3D places the complete center of the floor lamp below its visible part and correctly answers “floor lamp”. On D1000, SGT-3D meets the center, dimension and orientation tolerances for all queried objects in 85.8% of the questions, measured against full-object references (Table 3).