- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates how geometric generation can support spatial understanding, both as supervision during training and as explicit evidence during inference. The authors introduce GeoUG, built on BAGEL, with metric-depth and viewpoint generation capabilities. A multi-scale depth codec enables metric-depth prediction through the existing visual generation interface. For inference, a draft-conditioned policy learns whether to generate depth and obtain geometric measurements before retaining or revising its initial answer. The paper also introduces 3DCueBench, which balances selected 2D cues across geometric difficulty levels. Experiments show task-dependent benefits from generative co-training and improved single-image spatial reasoning with selective depth generation.
Critical strengths
- The paper presents a coherent approach connecting geometric supervision with inference-time geometric evidence. The draft-conditioned policy achieves accuracy close to always generating while reducing depth-generation calls, and its advantage over random selection supports the usefulness of the learned acquisition policy.
- 3DCueBench and the accompanying analysis provide a useful examination of spatial evaluation. Balancing selected cue-consistent and cue-conflicting examples across difficulty levels helps distinguish geometric discrimination from reliance on simple image-plane relationships.
Critical weaknesses
- The additional value of generative modeling over access to explicit geometric information remains insufficiently established. The experiments demonstrate that the proposed generation-assisted pipeline improves over direct answering. However, this comparison does not sufficiently distinguish the benefit of generating geometry from the benefit of obtaining geometric information. Non-generative alternatives could provide depth maps, numerical readouts, or geometry-encoder features to a comparably trained understanding model. This distinction matters because DA3 readout already achieves an average score of 92.8, compared with GeoUG’s 94.3, and essentially matches GeoUG on 3DCueBench. Appendix F.3 is relevant, but supplying measurements to existing VLMs does not provide a controlled comparison with a comparably trained geometry-augmented model; the appendix also acknowledges prompting and resolution differences. Consequently, the results establish a useful pipeline, but leave its incremental contribution relative to alternative forms of geometric integration unclear.
- The evidence for broader spatial understanding is limited to single-image reasoning and small sets of views. The strongest inference-time results concern single-image depth and distance comparisons. The co-training experiments on MindCube and the MMSI camera-motion subset provide some multi-view evidence, but do not establish the effectiveness of the generation-assisted reasoning procedure over substantially longer image sequences. Such settings introduce requirements beyond per-image depth quality, including cross-view alignment, geometric consistency, and integration of partially observed scene information. The relatively modest multi-view co-training gains do not resolve these questions. This limits the demonstrated breadth of the approach and leaves uncertainty about how far its benefits extend beyond the evaluated comparisons.
Questions for authors
- Could the authors provide a controlled comparison with a non-generative geometry-augmented model, using comparable spatial supervision and training conditions? For example, how does GeoUG compare with a model trained to consume depth maps, DA3 readouts, or geometry-encoder features and selectively use that information?
- How would the proposed generation-assisted reasoning procedure operate over longer sequences, such as 16–32 frames requiring cross-view spatial integration? Is there evidence that its benefits persist, and how would geometric inconsistencies across generated depth maps be handled?
- Given the strong DA3 readout results, what practical advantage does the generative route offer at comparable computational cost? Measurements of actual inference latency and training requirements would help assess whether the observed accuracy gains justify this route beyond reducing calls relative to always generating within GeoUG.
4 comments
Q2: Long-sequence reasoning and cross-view consistency
Thank you for raising this concern. We added 16-frame VSI-Bench experiments to test two distinct questions: whether geometric co-training transfers to video spatial understanding, and whether generated geometry can assist inference across multiple frames.
1. Training-time transfer to 16-frame spatial understanding. We compare four models, all of which answer directly from RGB frames without generating geometry at inference.
| Model | Avg. | Object counting | Object size | Absolute distance | Relative direction | Route planning |
|---|---|---|---|---|---|---|
| Original BAGEL | 31.99 | 23.88 | 33.75 | 24.65 | 45.73 | 31.96 |
| Understanding-only | 42.86 | 40.83 | 67.82 | 29.52 | 46.21 | 29.90 |
| + Multi-view generation | 43.99 | 40.85 | 70.21 | 29.51 | 47.41 | 31.96 |
| + Depth generation | 42.61 | 40.96 | 67.00 | 31.02 | 46.23 | 27.84 |
The task-dependent transfer observed in our original experiments extends to the 16-frame setting. Relative to understanding-only training, depth co-training improves absolute-distance estimation by 1.50 points, while multi-view co-training improves relative direction, route planning, and object size by 1.20, 2.06, and 2.39 points, respectively. Multi-view co-training also increases average performance from 42.86 to 43.99. These results support choosing geometric objectives according to the target spatial skill, rather than expecting every objective to improve every task.
2. Inference-time assistance with cross-view consistency checks. We compare direct and geometry-assisted reasoning on 88 geometry-eligible absolute-distance questions from 58 scenes, using the same GeoUG checkpoint and 16 RGB frames. Eligibility follows fixed grounding and geometric-support checks, not answer correctness.
GeoUG generates metric depth independently per frame using MMDC. Auxiliary object-region and camera-pose predictions align the corresponding surface points in a shared coordinate system. To handle inconsistencies between independently predicted depth maps, we apply occlusion-aware cross-view reprojection checks, retaining points supported by another depth map and rejecting ambiguous detections or insufficient geometric support. The model answers using the retained visible-surface distance estimates, without ground-truth depth or answer-based geometric correction.
| Inference condition on the same 88 questions | Absolute-distance MRA (×100) |
|---|---|
| Direct RGB reasoning | 31.82 |
| GeoUG-generated depth assistance | 60.45 |
The 28.63-point gain shows that generated geometry can assist 16-frame reasoning when combined with explicit alignment and consistency filtering. This result applies to the eligible subset in an externally orchestrated diagnostic, rather than a full-benchmark evaluation of an autonomous policy. Qualitative examples appear in Appendix A (Long-sequence inference).
Q3: Practical cost and benefit compared with DA3
Our goal is to understand how generation can help spatial understanding. GeoUG learns to generate geometric evidence and decide when to use it to answer spatial questions.
The DA3 readout baseline uses task-specific scripts to extract the supplied coordinates, compute depths or distances, and choose an answer from those values. In GeoUG, the model constructs the measurement queries and produces the answer. DA3 readout is a strong baseline for these structured tasks, but comparing the full reasoning systems would require a DA3-augmented model trained under comparable conditions.
GeoUG learns to generate depth with 6k additional training steps on 8 GPUs, starting from its pretrained generative backbone. DA3 starts from DINOv2 and reports 200k steps on 128 H100 GPUs, with DA3-Giant taking approximately 10 days. GeoUG also supports several geometric generation tasks within the same model, though we have so far tested their benefit to reasoning only through depth.
Our current implementation is substantially slower than DA3 readout. However, on 32 randomly sampled 3DCueBench questions, adaptive reasoning reduces mean latency from 63.28 s to 53.68 s, saving 9.60 s per request (15.2%) compared with forced generation using the same checkpoint, including drafting and decision overhead. We measured both modes on one A100-80GB GPU at batch size 1, with identical inference settings. After two warmup requests, each question was evaluated three times per mode using the original option order. Adaptive reasoning skips generation on 18.75% of requests, while both modes answer 89/96 requests correctly. This reduces GeoUG’s inference cost, but does not make it faster than DA3. We acknowledge this limitation and will report the absolute timings and full measurement protocol in the supplementary material.
Thank you for the additional experiments and clarifications. They address my main concerns, although the baseline comparison and multi-frame evaluation still have some limitations. I maintain my weak-accept recommendation.
Thank you for taking the time to review our additional experiments and clarifications. We’re glad they helped address your main concerns. We appreciate your constructive feedback, and we’ll take care to explain the remaining limitations more clearly in the revised manuscript. Thank you again!
Q1: Value of generative geometry
Thank you for distinguishing the value of generating geometry from that of accessing geometric information. We have added the requested trained, non-generative geometry-augmented baseline and evaluated it on all three benchmarks. This addresses a gap in our original evidence: direct answering and the off-the-shelf VLM comparisons in Appendix F.3 did not provide this trained comparison.
Our BAGEL–DA3 baseline uses frozen DA3 to estimate depth and returns numerical geometric measurements to the reasoning model. Spatial SFT trains the model to use these measurements, and RL trains it to acquire them selectively. We align the understanding initialization, main spatial questions and answer supervision, and RL recipe with GeoUG. This tests whether a trained model with access to external geometry can recover GeoUG's performance.
Full-benchmark circular accuracy (%); average across the three benchmarks.
1. GeoUG outperforms the trained external-geometry alternative, even when acquisition is forced. GeoUG exceeds adaptive BAGEL–DA3 by approximately 5.6 points on average. Forcing DA3 acquisition improves the baseline from 88.75% to 91.29%, but GeoUG still leads by approximately 3.0 points, with higher accuracy on every benchmark. The remaining gap therefore cannot be attributed solely to the baseline choosing not to acquire geometry. This comparison strengthens the evidence beyond an improvement over direct answering: GeoUG also outperforms a model explicitly trained to access and use external geometric measurements.
2. The strong deterministic DA3 result helps distinguish geometry quality from learned use of geometry. We retain this baseline prominently: its 92.8% average exceeds both trained BAGEL–DA3 variants, and GeoUG's advantage over it is 1.5 points, with essentially matched performance on 3DCueBench. Thus, our claim is not that DA3 lacks useful geometric information. Rather, access to that information does not by itself reproduce GeoUG's performance in the evaluated learned reasoning pipeline. The deterministic readout and trained baseline answer different questions: the former measures what task-specific rules can extract from DA3, while the latter tests a model trained to acquire and reason with its measurements.
3. The demonstrated advantage belongs to the complete integration, rather than to geometry generation in isolation. GeoUG retains a VAE-encoded representation of its generated depth alongside numerical query results; the evaluated BAGEL–DA3 baseline receives numerical measurements alone. GeoUG therefore provides the reasoning model with both a depth representation and explicit measurements. This contextual representation, together with joint generation–understanding training, offers a plausible explanation for the performance gap, but the current comparison does not isolate their individual effects. We will make this input distinction explicit rather than attribute the entire gain to the generative depth source.
The new experiment supports a concrete conclusion: under the aligned training settings described above, the complete GeoUG system outperforms the evaluated trained non-generative numerical-tool alternative, including when that alternative always acquires geometry.