acceptodds
Under review as a conference paper at ICLR 2027

StereoVLM: Binocular Vision-Language Models for Metric 3D Spatial Reasoning

34 views

Abstract

Vision-language models (VLMs) exhibit strong semantic understanding but remain limited in their ability to reason about metric 3D space in the physical world. Existing VLMs typically infer 3D scene structure from one or more images without known camera calibration or cross-view geometric constraints. This makes metric 3D inference inherently ambiguous and leaves it unclear whether their spatial predictions are grounded in visual evidence or driven primarily by semantic priors. In contrast, calibrated stereo provides explicit cross-view geometry that resolves this ambiguity, offering a more reliable foundation for metric 3D perception and geometric generalization. We investigate how VLMs can learn transferable 3D representations directly from calibrated binocular observations. To this end, we introduce **StereoVLM**, the first geometry-aware binocular VLM for metric 3D spatial reasoning. **StereoVLM** follows a hierarchical learning paradigm spanning from cross-view correspondence and disparity estimation to calibration-conditioned metric depth perception and semantic spatial reasoning. We further find that disparity supervision generalizes substantially better than direct binocular depth supervision on unseen environments. To evaluate these capabilities, we introduce **Stereo3D-Bench**, a comprehensive benchmark across three levels, from point-level geometry to object-level spatial reasoning. **StereoVLM-8B** achieves 95.12% depth accuracy () and a 66.98 average object-level score on Stereo3D-Bench. Binocular training yields average gains of 3.54 points across five spatial benchmarks and 4.3 percentage points in RoboCasa manipulation success. Intervention results further indicate that these gains depend on valid cross-view geometry and cannot be explained solely by additional visual tokens or semantic priors.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language Models, Stereo Vision, 3D Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PmFiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces StereoVLM, a geometry-aware binocular vision-language model for metric 3D spatial reasoning, which learns transferable 3D representations from calibrated binocular observations. Initialized from Qwen3-VL-Instruct, the model undergoes mixed-task instruction tuning with hierarchical supervision spanning cross-view correspondence, disparity estimation, calibration-conditioned metric depth perception, oriented 3D grounding, and task-target grounding. The authors also introduce Stereo3D-Bench, which comprises eight tasks across three levels: point-level geometry, object-level geometry, and object-level spatial reasoning. StereoVLM-8B achieves a point-level depth δ1\delta_1 score of 95.12% and an average score of 66.98 across the five object-level tasks. Additional experiments with StereoVLM-2B demonstrate transfer gains on external spatial-reasoning benchmarks and downstream RoboCasa manipulation tasks.

Critical strengths
  • Well-motivated geometry-consistent supervision. Geometry-Consistent Stereo Supervision treats cross-view correspondence, disparity, and metric depth as complementary supervision targets derived from the same stereo geometry. Table 1 shows that depth recovered from binocular disparity achieves a δ1\delta_1 score of 74.92% on unseen scenes, substantially outperforming direct binocular depth supervision at 45.89%.

  • Broad evaluation with informative ablations and diagnostic controls. Stereo3D-Bench extends evaluation from point-level geometry to language-referred objects and task-target grounding. The authors further conduct controlled right-view interventions, including single-left, zero-right, duplicated-left, and shuffled-right inputs, while keeping the checkpoint, question, calibration, and decoding procedure fixed.

Critical weaknesses
  • Limited comparison with modular stereo-based approaches. The main comparisons focus on general-purpose VLMs and monocular depth VLMs, without evaluating a modular pipeline combining a specialized stereo matching model with a VLM. This comparison would help clarify StereoVLM’s advantages over an explicit geometry-based pipeline.
Questions for authors

I am not very familiar with this research area, and the following questions would help me better understand the paper.

  1. Benefits of joint learning. Could StereoVLM be compared with a modular pipeline that uses a stereo estimator, such as FoundationStereo [1], to provide geometry to a VLM?

  2. Generalization. The held-out in-house benchmark evaluates unseen-scene generalization, while the calibration perturbation experiment assesses robustness to residual extrinsic errors. Could the authors briefly clarify whether camera parameters, such as focal length and stereo baseline, differ between the training and test data, and whether any such variation is covered by the existing evaluation?

[1] Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

3 comments

We thank Reviewer #PmFi for the constructive feedback and recognition of our supervision design and evaluation. We address your concerns below.

Comparison with modular stereo-vlm pipelines

Table 17. Accuracy and time efficiency of modular stereo–VLM pipelines and StereoVLM-2B. Both modular pipelines use FoundationStereo (FS) and frozen Qwen3-VL-2B. Latency includes FoundationStereo inference.

ModelAvg.Corr.Disp.Depth3D BoxTargetMean latency (s)
Localization-based35.8143.7539.7077.6712.805.132.212
Direct-input5.077.510.5217.030.310.000.920
StereoVLM-2B64.4386.0653.9297.3936.0648.720.643

We thank the reviewer for this suggestion. We have added comparisons with two modular pipelines combining FoundationStereo and a frozen Qwen3-VL-2B-Instruct (Appendix D.7, Table 17):

  • Localization-based: The VLM first localizes the queried object in the left image. Geometric measurements are then extracted from FoundationStereo predictions within the localized region and provided to the VLM to generate the final answer.
  • Direct-input: The VLM receives stereo images, a predicted depth visualization, a disparity/depth grid, and camera calibration in a single call.

StereoVLM-2B outperforms both pipelines on all five semantic tasks in Stereo3D-Bench, achieving an average score of 64.43, compared with 35.81 and 5.07 for the localization-based and direct-input pipelines, respectively. Its mean end-to-end latency is 0.643 s, versus 2.212 s and 0.920 s, corresponding to 3.44× and 1.43× speedups. Both modular latency measurements include FoundationStereo inference.

Compared with these explicit geometry pipelines, StereoVLM directly predicts language-referred object geometry and task targets from stereo images and the task prompt in one VLM call, without a separate stereo estimator or intermediate geometry extraction and encoding. These results demonstrate the practical advantages of the integrated approach in both semantic accuracy and inference efficiency.

Generalization Across Camera Configurations

We clarify the camera configurations used in training and evaluation in the table below.

Table: Camera configurations used in training and evaluation.

Data sourceTraining baseline (cm)Test baseline (cm)
DROID11.975–12.01711.975–12.017
In-house Robot (training source)5.773–
RoboCasa6.0006.000
Xperience-10M14.498–14.87914.500–14.879
In-house Robot (held-out source)–6.346–6.430
  • Focal length. Our preprocessing maps stereo images to a common focal length of 267 px across all datasets, so both training and test images use a standardized focal length. For clarity, we have also added this explanation to Section 4.1 of the revised manuscript.
  • Baseline and evaluation coverage. We retain the calibrated baseline associated with each stereo pair where available and provide it in metric-depth prompts. The DROID, RoboCasa, and Xperience-10M test baselines overlap with those used in training. The held-out in-house benchmark uses baselines of 6.35–6.43 cm, which differ from those of the calibrated training sources while remaining within the overall training range. Thus, the existing held-out evaluation covers both unseen scenes and a different baseline configuration within this range.

To further examine how predictions respond to the supplied baseline, we have added a sensitivity experiment on 400 stereo pairs: 200 from DROID and 200 from our in-house robot dataset (Appendix D.4, Figure 8). Keeping the images, query pixels, focal length, and model checkpoint fixed, we vary only the baseline in the prompt from 0.6 to 1.5 times its nominal value. The median predicted depth ratios broadly follow ideal scaling in both domains, suggesting that the model uses the supplied baseline for metric depth prediction.

I appreciate the authors' efforts for the rebuttal phase. Your responses have addressed my concerns, so I will maintain my positive score.

Thank you for your positive feedback and for maintaining your supportive assessment of our work. We are glad that our responses have addressed your concerns, and we sincerely appreciate your time and thoughtful consideration.

Reviewer_NfWTOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I think StereoVLM fine-tunes Qwen3-VL on calibrated stereo pairs to answer metric 3D questions. Training mixes point-level correspondence, disparity and calibrated depth questions with object-level depth, 3D box and task-target grounding, using FoundationStereo labels. It also introduces Stereo3D-Bench and reports that disparity supervision generalizes much better than direct binocular depth on an unseen in-house set (74.92 vs 45.89 δ1, Table 1). StereoVLM-8B reaches 95.12 δ1 and a 66.98 object average (Table 4). Stereo tuning adds 3.54 points on average across five external spatial benchmarks (Table 6) and 4.3 points of RoboCasa success as a VLA backbone (Table 7).

Critical strengths

I think the best evidence is the right-view intervention suite in Table 5, on one fixed checkpoint and prompt. Going from single-left to correct stereo lifts depth δ1 from 80.82 to 94.51 and the object average from 40.26 to 64.43, and duplicated-left collapses to 7.40, which is what you'd expect if zero disparity reads as far depth. That convinces me the model really uses the second view, the paper's core premise, though only at 2B.

I also like that Stereo3D-Bench is scored against source-native geometry rather than the FoundationStereo teacher, with scene-grouped splits (App. A-B). If it's released, I suspect people will keep using it.

Critical weaknesses

My main concern is that the transfer claim isn't isolated. The abstract says the gains "depend on valid cross-view geometry", but the Table 6/7 control just drops the stereo data (p.9), so the +stereo arm also gets extra metric-depth and 3D box labels. There's no left-only arm with the same labels, and none of the five external benchmarks has a stereo pair at test time (Omni3D alone gives 8.21 of the 17.69 summed gain). On RoboCasa, D.4 admits the duplicated-left drop (42.4 to 26.3) "cannot distinguish geometric dependence from other effects", and without stereo pretraining the single-left policies beat the stereo ones (36.4 vs 34.9, 39.5 vs 38.1, Table 14). So I read Table 6 as "more 3D labels help", which isn't the abstract's claim.

I'm also not convinced Table 4 measures binocular reasoning rather than fine-tuning. Every baseline is zero-shot, while StereoVLM trains on DROID, FSD and Xperience with the benchmark's own templates. With no right image, StereoVLM-2B already gets 80.82 δ1 (Table 5), above GPT-6 Astra with stereo (75.96). I think the fair binocular increment is about 13 points (94.51 vs 80.82; 91.07 vs 77.91 in Table 1), not the "+58.01 over DepthVLM-4B" on p.8. There's no fine-tuned monocular or stereo-network row either, and since Eq. 6 keeps DROID and Xperience pixels only where S2M2 is within 10% of the reference, S2M2 would trivially score 100% δ1 there. Also, the p.8 prose (4.11/94.96/65.55) doesn't match Table 5 (4.09/94.51/64.43).

Questions for authors
  1. Could you add a left-only arm trained on the same depth and 3D box labels for Tables 6 and 7? I think that's the run most likely to change my score.
  2. How would a fine-tuned monocular model and S2M2 do in Table 4, and what are the per-domain numbers, especially RoboCasa?
  3. I'm curious about Table 1's unseen set: its size, rig and GT, and whether it's independent of the in-house training collection (p.6 l.309). Correct me if I'm wrong, but the disparity variant gets Z = fxB/d for free while the depth variant has to learn it from text, so how much of the 29.03-point gap is just that?
  4. Does depth really use the stated calibration? Dropping B costs 7.31 δ1 (80.82 vs 73.51, Tables 5 and 12), but D.3 only perturbs the right-camera pose. I wonder what happens if you vary fx or B.
4 comments

We sincerely thank Reviewer #NfWT for the thoughtful and constructive comments and suggestions. We appreciate your recognition of our right-view intervention experiments and benchmark design. We will open-source this benchmark and provide a more complete version with more data sources in the future. We address your concerns point by point below.

Disentangling spatial supervision from binocular geometry

  • Controlled transfer study on spatial-reasoning benchmarks.

Table 14: Monocular(Omni3D, DA-2K, OmniSpatial) and Video(MindCube, VSI-Bench) Spatial Reasoning by Training Input and Geometric Supervision.

Training inputCorr.Disp.DepthOmni3DDA-2KOmniSpatialMindCubeVSI-BenchAvg. ↑
w/o stereo–––23.9573.7940.1844.8656.9447.94
Correct stereo (Iₗ, Iᵣ)––✓30.0971.7644.1047.8157.4050.23
Single left Iₗ––✓28.5080.6640.7045.8155.3950.21
Duplicated left (Iₗ, Iₗ)––✓32.7875.5841.8147.0550.0849.46
StereoVLM-2B✓✓✓32.1676.1642.6648.5757.8651.48 (+3.54)

The original ablation study in Section 5.4 measures the contribution of the overall binocular instruction set, but does not disentangle the effects of additional spatial supervision from those of valid binocular geometry. To further investigate these two factors, we added a controlled comparison of the following four training variants in Appendix D.5.1, Table 14 of the revised manuscript:

  • Correct Stereo: Uses correctly paired stereo images with supervision for depth estimation and object-/target-level 3D localization.
  • Single Left: Uses only the left image, with the same supervision as Correct Stereo.
  • Duplicated Left: Duplicates the left image across both input positions, with the same supervision as Correct Stereo.
  • StereoVLM: Uses correctly paired stereo images and adds correspondence and disparity supervision to the supervision used by Correct Stereo.

To ensure that the first three input configurations share the same supervision set, we removed correspondence- and disparity-related questions from all three, because a single-left-image input lacks the right-view observations required for these tasks. Therefore, these three variants use a reduced supervision set to isolate the effect of the training input, whereas StereoVLM is used to assess the additional benefit of full geometric supervision.

As shown in Table 14, Correct Stereo, Single Left, and Duplicated Left achieve average scores of 50.23, 50.21, and 49.46, respectively, all outperforming the baseline without stereo instruction tuning (47.94). This result indicates that additional spatial supervision itself provides transferable benefits. Meanwhile, the comparable average performance of Correct Stereo and Single Left suggests that merely providing correctly paired stereo images is insufficient to guarantee better transfer performance. Correct Stereo outperforms Duplicated Left by 0.77pp, indicating that valid cross-view information offers additional value when the supervision and two-image input format are held constant, although this advantage is not consistent across all benchmarks. Single Left performs best on DA-2K, a monocular relative-depth benchmark, which is consistent with its matched single-image training and evaluation setting.

StereoVLM, trained with full geometric supervision, achieves the highest average score of 51.48, outperforming Correct Stereo by 1.25pp and obtaining the best results on both video benchmarks, MindCube and VSI-Bench. These findings support our design: spatial supervision itself provides transferable benefits, while jointly learning binocular geometry through correspondence, disparity, and depth supervision further improves overall transfer performance.

  • Controlled transfer study on downstream manipulation.

Table 15: Manipulation Success (%) by Visual Input and Geometric Supervision (Corr./Disp./Depth). Each configuration uses the same visual inputs during training and evaluation. The gain is relative to the best depth-supervised variant.

Visual inputCorr.Disp.DepthSuccess ↑
Correct stereo (Iₗ, Iᵣ)––✓38.2
Single left Iₗ––✓37.7
Duplicated left (Iₗ, Iₗ)––✓36.3
StereoVLM-2B✓✓✓42.4 (+4.2)

Table 15 (Appendix D.5.2) presents controlled manipulation experiments. With depth-only supervision, correct stereo and single-left inputs yield similar success rates (38.2% vs. 37.7%). Adding correspondence and disparity supervision improves success to 42.4% (+4.2pp). Consistent with the spatial-reasoning results, this suggests that stereo inputs alone are insufficient, while explicit cross-view geometric supervision improves downstream manipulation.

Comparison with S2M2 and per-domain evaluation.

We compare with a fine-tuned monocular baseline and S2M2, reporting per-domain depth accuracy. The monocular baseline uses left-image depth and 3D spatial QA with the same target labels, but without the right image or stereo baseline.

We report point-level depth δ₁ (%). S2M2's 100% on its filtered subsets is expected by construction. Compared with the monocular baseline trained with identical target labels, StereoVLM-2B gains 4.53pp on Stereo3D-Bench and 17.34pp on OOD data, showing benefits beyond task-specific fine-tuning.

ModelDROIDRoboCasaXperience-10MAvg. on Stereo3D-BenchOOD
Monocular depth-only (2B)89.1798.1882.6189.9851.58
S2M210098.610099.5497.03
StereoVLM-2B94.8698.4890.1794.5168.92
StereoVLM-8B95.5298.5291.4795.1770.16

Clarifying the unseen evaluation set and the role of analytical depth conversion.

Evaluation set and independence. The unseen set in Table 1 contains 500 stereo pairs with 120 evaluated points each (60,000 points). It comes from an entirely separate data source, independently collected using different scenes and capture hardware from the training data. No scenes, images, or derived annotations from this set were used for training.

The data was captured using PICO 4 Ultra stereo cameras with a TOF depth sensor. Before rectification, the calibrated intrinsics, expressed at the original per-eye image resolution of 2048×1536, have the following ranges:

Table: Camera intrinsics before rectification.

Intrinsic parameterLeft camera (px)Right camera (px)
fxf_x932.60–938.74931.19–939.38
fyf_y922.04–928.47920.45–929.33
cxc_x1012.69–1027.831010.03–1028.23
cyc_y767.42–784.15761.79–776.30

The calibrated stereo baselines range from 6.346 to 6.430 cm. Our preprocessing rectifies the images to 800×320, with fₓ=fy=267 px and principal point (400,160), while retaining the calibrated baseline for each pair.

Reference geometry. We construct the reference geometry using the same procedure as for the DROID and Xperience-10M training data, with reference metric depths obtained from sensor measurements.

Role of analytical depth conversion. We agree with the reviewer that explicitly applying Ẑ=fₓB/d̂ can contribute to the advantage observed in Table 1. This is an intentional part of the formulation being evaluated. Both pipelines use the same stereo images and camera calibration; the direct-depth variant also receives fₓ and B in its prompt.

The key distinction is how the prediction problem is structured. Direct depth supervision asks the model to learn metric depth from the images and calibration, without explicitly supervising the intermediate cross-view displacement. Disparity supervision directly targets this displacement, while the known geometric relation handles its conversion to metric depth. The formula does not provide the correct disparity or repair disparity estimation errors; The model must still infer the underlying visual geometry. This separation provides a plausible explanation for the stronger OOD performance of the disparity-based approach.

Sensitivity to focal length and baseline.

We have added a baseline sensitivity experiment (Appendix D.4, Figure 8) and clarified our focal-length preprocessing below.

  • Focal length. Our preprocessing maps input images to a common focal length, following the focal-length unification principle also used in VLM³ (see its Appendix A.1). The model therefore receives inputs with a standardized focal length. For clarity, we have also added this explanation to Section 4.1 of the revised manuscript.
  • Baseline. We evaluate the response of directly predicted depth to the supplied baseline on 400 stereo pairs: 200 from DROID and 200 from our in-house robot dataset. We keep the stereo images, eight query pixels per pair, focal length, and model checkpoint fixed, and modify only the baseline in the prompt to B'=α B, where α∈{0.6,0.8,0.9,1.0,1.1,1.2,1.5}. With image disparity fixed, ideal stereo geometry predicts Ẑ(B')/Ẑ(B)=α. For each domain, we report the median predicted depth ratio and its 10th–90th percentile range over all 1,600 query points, without filtering. The median ratios broadly follow ideal scaling in both domains, suggesting that the model uses the supplied baseline for metric depth prediction.

Correction of numerical inconsistencies.

We thank the reviewer for identifying this inconsistency. We have corrected the prose on page 8 and the corresponding entry in Table 5. For StereoVLM-2B under correct stereo input, the correct results are 4.12 px disparity MAE, 94.51% depth δ₁, and a 64.43 object-level average, consistent with Table 4. The prose inadvertently retained values from an earlier manuscript version.

Thank you for the response and the new controls as I suggest. The left-only arms, the monocular and S2M2 comparisons, and the unseen-set details address my main concerns. Also please narrow the transfer claim to geometric supervision rather than stereo input alone, as Table 14 suggests. I think will raise my score accordingly.

Thank you for your positive feedback and helpful suggestions. We have revised the abstract, introduction, transfer analysis, and conclusion to clarify that the transfer gains are supported by geometric supervision rather than stereo input alone, consistent with Table 14. Thank you once again for your insightful comments. We welcome any further questions or suggestions you may have.

Reviewer_SxFfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StereoVLM trains a visual language model with binocular geometric capabilities using correspondence points, parallax, camera-calibrated depth, and 3D localization commands for objects and task targets. It also proposes a Stereo3D-Bench covering eight tasks. The paper compares the generalization ability of different geometric supervisions, uses right-view intervention to test whether the model uses matched binocular evidence, and evaluates transfer performance on external space reasoning and RoboCasa manipulation tasks.

Critical strengths
  1. The paper presents valuable and well-controlled empirical findings. Section 5.2 compares three types of supervision under the same 234K image pairs and training budget. In Table 1, disparity supervision achieves a depth accuracy of 74.92% in unseen scenes, significantly higher than the 45.89% of direct binocular depth supervision. This is more insightful than simply showing that binocular is superior to monocular, demonstrating that the choice of geometric targets substantially affects generalization.

  2. The evaluation not only compares final accuracy but also examines the model's dependence on geometric evidence. Table 5, with the model, problem, and calibration fixed, changes the content of the right image; correct binocular input outperforms single left image, empty right image, and mismatched right image, supporting the benefit from effective pairing information. Table 6 reports improvements on all five external benchmarks under the same training budget, and Table 7 further demonstrates the benefits on operational tasks, making the contribution exceed the self-built benchmark.

Critical weaknesses
  1. External transfer benefits cannot yet be solely attributed to binocular geometric learning. The comparison in Section 5.4 removes the entire binocular instruction set and reallocates the sampling ratio of the remaining data, thus simultaneously changing the binocular information and geometric/object-level supervision content in the input. Table 5 demonstrates that the trained model relies on the right-hand image for this benchmark, and Table 3 indicates that point-level supervision is beneficial. However, neither result rules out the possibility that the performance gains shown in Table 6 stem primarily from the addition of spatial task supervision. It is necessary to further disentangle the benefits derived from the overall training recipe versus those contributed by binocular geometry itself.

  2. The positioning relative to the most closely related work is not entirely accurate. Section 2 characterizes VLM3 and other methods as focusing primarily on monocular point-level or pixel-level perception; yet, Section 3.2 of the VLM3 paper already incorporates cross-image pixel correspondences, camera poses, and object-level 3D understanding. While this is not identical to the calibrated binocular joint supervision proposed in this paper, the distinction must be clarified—otherwise, the uniqueness of the contribution (linking point-level geometry to semantic spatial understanding) might be overstated. A more robust basis for distinguishing this work lies in the joint learning of disparity and calibrated depth, along with the associated controlled experiments.

Questions for authors
  1. Are there any ablation studies—using the same images, comparable spatial labels, and an equivalent training budget—that isolate the impact of binocular inputs or disparity supervision, thereby distinguishing the benefits of the supervision content from those of the binocular geometry itself?

  2. Please clarify the differences between this work and the multi-view correspondence and object-level 3D training tasks found in VLM3—specifically, what unique capabilities are provided by the joint training approach versus the inclusion of calibration information?

2 comments

We sincerely thank Reviewer #SxFf for your thoughtful and constructive comments and suggestions. We appreciate your recognition of our well-controlled empirical findings on geometric supervision and generalization, as well as our evaluation of binocular evidence use and downstream transfer. We address your concerns point by point below.

Disentangling spatial supervision from binocular geometry

  • Controlled transfer study on spatial-reasoning benchmarks.

Table 14: Monocular(Omni3D, DA-2K, OmniSpatial) and Video(MindCube, VSI-Bench) Spatial Reasoning by Training Input and Geometric Supervision.

Training inputCorr.Disp.DepthOmni3DDA-2KOmniSpatialMindCubeVSI-BenchAvg. ↑
w/o stereo–––23.9573.7940.1844.8656.9447.94
Correct stereo (Iₗ, Iᵣ)––✓30.0971.7644.1047.8157.4050.23
Single left Iₗ––✓28.5080.6640.7045.8155.3950.21
Duplicated left (Iₗ, Iₗ)––✓32.7875.5841.8147.0550.0849.46
StereoVLM-2B✓✓✓32.1676.1642.6648.5757.8651.48 (+3.54)

The original ablation study in Section 5.4 measures the contribution of the overall binocular instruction set, but does not disentangle the effects of additional spatial supervision from those of valid binocular geometry. To further investigate these two factors, we added a controlled comparison of the following four training variants in Appendix D.5.1, Table 14 of the revised manuscript:

  1. Correct Stereo: Uses correctly paired stereo images with supervision for depth estimation and object-/target-level 3D localization.
  2. Single Left: Uses only the left image, with the same supervision as Correct Stereo.
  3. Duplicated Left: Duplicates the left image across both input positions, with the same supervision as Correct Stereo.
  4. StereoVLM: Uses correctly paired stereo images and adds correspondence and disparity supervision to the supervision used by Correct Stereo.

To ensure that the first three input configurations share the same supervision set, we removed correspondence- and disparity-related questions from all three, because a single-left-image input lacks the right-view observations required for these tasks. Therefore, these three variants use a reduced supervision set to isolate the effect of the training input, whereas StereoVLM is used to assess the additional benefit of full geometric supervision.

As shown in Table 14, Correct Stereo, Single Left, and Duplicated Left achieve average scores of 50.23, 50.21, and 49.46, respectively, all outperforming the baseline without stereo instruction tuning (47.94). This result indicates that additional spatial supervision itself provides transferable benefits. Meanwhile, the comparable average performance of Correct Stereo and Single Left suggests that merely providing correctly paired stereo images is insufficient to guarantee better transfer performance. Correct Stereo outperforms Duplicated Left by 0.77pp, indicating that valid cross-view information offers additional value when the supervision and two-image input format are held constant, although this advantage is not consistent across all benchmarks. Single Left performs best on DA-2K, a monocular relative-depth benchmark, which is consistent with its matched single-image training and evaluation setting.

StereoVLM, trained with full geometric supervision, achieves the highest average score of 51.48. These findings refine our original claim: spatial supervision provides substantial transferable benefits, while adding explicit correspondence and disparity supervision on top of depth and spatial supervision yields a further 1.25-point average gain and the best performance on both video benchmarks.

  • Controlled transfer study on downstream manipulation.

Table 15: Manipulation Success (%) by Visual Input and Geometric Supervision (Corr./Disp./Depth). Each configuration uses the same visual inputs during training and evaluation. The gain is relative to the best depth-supervised variant.

Visual inputCorr.Disp.DepthSuccess ↑
Correct stereo (Iₗ, Iᵣ)––✓38.2
Single left Iₗ––✓37.7
Duplicated left (Iₗ, Iₗ)––✓36.3
StereoVLM-2B✓✓✓42.4 (+4.2)

Table 15 (Appendix D.5.2) presents controlled manipulation experiments. With depth-only supervision, correct stereo and single-left inputs yield similar success rates (38.2% vs. 37.7%). Adding correspondence and disparity supervision improves success to 42.4% (+4.2pp). Consistent with the spatial-reasoning results, this suggests that stereo inputs alone are insufficient, while explicit cross-view geometric supervision improves downstream manipulation.

Clarifying the relationship to VLM3

The distinctions concern both the visual input setting and the organization of supervision, as detailed below.

  1. Monocular versus binocular metric and object-level understanding. VLM³ studies metric depth estimation and object-level 3D understanding with single-image inputs, while its multi-view experiments address cross-image correspondence and camera pose estimation. StereoVLM instead studies metric perception and language-grounded object understanding from calibrated binocular observations. It uses stereo pairs for point- and object-level correspondence, disparity, and metric depth, as well as oriented 3D grounding and manipulation-target grounding. Thus, our distinction is not simply the inclusion of multiple images, but the use of calibrated binocular evidence across both geometric and semantic spatial tasks.
  2. Joint learning within a single model. VLM³ reports task-specific training for depth estimation, cross-image correspondence, camera pose estimation, and object-level 3D understanding using a common methodology. StereoVLM jointly learns binocular geometric perception and language-grounded spatial understanding within a single model. Its eight tasks cover point- and object-level correspondence, disparity, and metric depth, together with language-referred oriented 3D grounding and manipulation-target grounding. Our distinction lies in jointly learning these geometrically related tasks and studying their generalization and transfer, rather than claiming that the individual tasks are new.

We agree that VLM³ studies cross-image correspondence and monocular object-level 3D understanding as separate task settings. We have revised Section 2 to describe its scope more accurately. Unlike VLM³, StereoVLM jointly learns point-level binocular geometry and language-grounded object-level 3D understanding from calibrated stereo inputs within a single model.

The roles of joint supervision and calibration

  1. Joint supervision provides complementary training signals across geometric and semantic levels. Table 3 evaluates this connection by removing point-level geometry QA from training. Including this supervision improves all five object-level metrics, including object-depth δ₁ from 91.52% to 97.39% and oriented 3D-box mIoU from 26.24% to 36.06%. These results support the benefit of incorporating point-level geometric supervision into language-grounded object-level learning.
  2. Calibration relates image-space geometry to physical scale. Focal length and baseline connect disparity to metric depth through Z=fₓB/d, while camera intrinsics relate depth and pixel locations to metric 3D coordinates. In Appendix D.4 (Figure 8) of the revised manuscript, we examine whether the model uses the supplied calibration by varying only the baseline in the prompt while keeping the images, query pixels, focal length, and model checkpoint fixed. The directly predicted depth ratios broadly follow the expected baseline scaling, supporting the model’s use of the supplied baseline for metric-depth prediction.

These components therefore have complementary roles: joint supervision links geometric estimation with semantic spatial tasks, while calibration supplies the physical scale relationship. VLM³ also uses intrinsics for focal-length unification; our distinction is how calibration is integrated with jointly supervised binocular tasks, rather than calibration use alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.