acceptodds
Withdrawn from ICLR 2027

RoboStereo: Recurrent 4D Reconstruction from Stereo Videos for Embodied Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Embodied robots require accurate 3D geometry during close-range interactions, yet object motion and limited relative viewpoint changes can leave local geometry weakly constrained in monocular video reconstruction. Synchronized stereo provides geometric evidence, but textureless regions, repetitive patterns, and occlusion can make matching ambiguous. We present RoboStereo, a recurrent model that reconstructs per-frame dense geometry and cameras in a shared coordinate system from synchronized, rectified stereo videos. Within a single inference run, multi-frame representations repeatedly guide stereo matching, while matching features feed back into the reconstruction state. To address the limited scene diversity of existing stereo data, we render annotated stereo videos for training and evaluation and construct the Embodiment Stereo Benchmark from selected renderings, with an emphasis on dynamic scenes and embodied manipulation. Experiments on our benchmark and public datasets demonstrate improved scale-aligned depth estimation over stereo and monocular multi-frame baselines, including on manipulation datasets excluded from joint training. Qualitative comparisons further show clearer local geometry around hand–object interactions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D reconstruction, stereo video, dynamic scene reconstruction, embodied perception, recurrent refinement

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o8bsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RoboStereo, which performs 4D reconstruction — per-frame depth, camera pose and world point maps — from stereo video, targeting embodied/robotic settings where a calibrated stereo rig is available and metric scale matters. The core mechanism is Geometry-Guided Stereo Fusion, which interleaves a DVLT-style loop-transformer multi-frame reconstruction backbone with WAFT-Stereo-style disparity refinement. A second contribution is data: eleven rendered/synthetic training sources with stated sampling weights, plus the Embodiment Stereo Benchmark.

RoboStereo achieves the lowest scale-aligned depth AbsRel on all seven benchmark groups, the best delta-1/RMSE on five of seven, and the best camera ATE/translational RPE on four of seven.

Critical strengths

(a) A coupled, rather than one-way, integration of stereo matching and multi-frame reconstruction. The bidirectional recurrent loop is well matched to the stated problem (local manipulation geometry is weakly constrained by monocular multi-frame aggregation), and the read/write ablation shows each direction is useful and that the full model beats the baseline (one-way WAFT-Stereo -> Pi3X) on every group.

(b) A data and benchmark contribution targeting an underexplored niche. The rendered collection covers close-range and wrist-mounted embodied interactions absent from Dynamic Replica-style datasets, and the benchmark protocol includes groups (e.g., EBench wrist) excluded from joint training, which partially guards against in-domain overfitting and gives the community a reusable evaluation resource.

Critical weaknesses

(a) No quantitative real-world evaluation. Every scored result is on synthetic renderings from simulator families also used for training; real stereo footage is shown only qualitatively. For an embodied vision system this is a major gap: real stereo failures arise from sensor noise, rectification error, exposure changes, and motion blur, and the metric rescaling assumes a constant focal-baseline product that physical rigs only approximate. The central practical claim therefore remains unverified.

(b) Limited novelty beyond combining two existing systems, and uneven camera results. The architecture is essentially the DVLT backbone plus WAFT-Stereo with gated adapters, the authors explicitly state they cannot separate gains due to joint modeling from gains due to the rendered data, and camera accuracy is best on only four of seven groups (rotational RPE is rarely best). There is also no comparison with Stereo4D, which targets the same dynamic-stereo reconstruction setting. These points weaken both the novelty and the "geometry plus cameras" claims.

Questions for authors

(1) Can you provide quantitative results on real stereo data (e.g., against RGB-D or scanned-scene GT), including robustness to injected rectification error and noise, since metric rescaling depends on a constant f*b?

(2) Why is Stereo4D not included, and how does the model behave on internet or real stereo video rather than simulator renderings?

(3) Please train the strongest baseline (WAFT-Stereo, or WAFT-Stereo -> Pi3X) on the same eleven-source mixture with the same budget. How much of the margin is data and how much is architecture?

Reviewer_Wy43OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RoboStereo, a framework for 4D reconstruction from stereo videos. It combines a recurrent multi-frame reconstruction backbone with stereo matching through bidirectional feature exchange, jointly predicting depth, disparity, camera poses, and 3D geometry. The authors also construct a synthetic stereo benchmark covering different embodied environments. Experiments show improved depth estimation and competitive reconstruction performance across several datasets.

Critical strengths

The idea of combining stereo matching with multi-frame reconstruction is reasonable. The bidirectional feature exchange allows temporal context to improve stereo matching and stereo information to refine the reconstruction. The ablation results support this design.

The experimental evaluation covers a variety of datasets, including challenging wrist-mounted views and manipulation scenes. The reported depth improvements are substantial on several benchmarks.

Critical weaknesses

The baseline comparisons overlook two highly relevant works, FoundationStereo (CVPR 2025) and Fast-FoundationStereo (CVPR 2026). Both are strong zero-shot stereo matching methods, yet neither is discussed or evaluated. Without these comparisons, it is difficult to assess the actual performance of RoboStereo against existing SOTA stereo methods.

The claim of 4D reconstruction is not fully supported. Most evaluations focus on per-frame depth and camera estimation, with limited analysis of temporal consistency, dynamic object motion, or interaction reconstruction.

The quantitative evaluation is limited to synthetic datasets, despite the availability of real-world stereo benchmarks with ground-truth disparity, such as Middlebury, ETH3D, and KITTI. Furthermore, the main depth results rely on ground-truth-based scale alignment, leaving the accuracy of metric depth estimation unclear.

Questions for authors

Why are FoundationStereo and Fast-FoundationStereo missing from both the related work and experimental comparisons?

How much of the improvement comes from the proposed architecture rather than the additional synthetic training data?

How does the method improve temporal consistency and dynamic 4D reconstruction beyond per-frame depth estimation?

What is the inference time and memory consumption of RoboStereo compared to existing stereo methods?

Reviewer_xQpHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

A recurrent model that reconstructs per-frame depth and cameras from rectified stereo video. It interleaves a DVLT backbone with WAFT-Stereo-style disparity updates through gated bidirectional feature exchange. The authors also render stereo data from eight simulated sources and build a 7-group Embodiment Stereo Benchmark. Validation covers scale-aligned depth (best AbsRel on all 7 groups and on 4 of 6 public sets), camera pose, point clouds, and a read/write ablation.

Critical strengths
  • Large gains where baselines fail. It has the best AbsRel on all groups, with big wrist-view gains (RoboCasa wrist 0.0296 vs 0.1275; held-out EBench wrist 0.1371 vs 0.1717). This is the regime the paper targets.
  • Bidirectional fusion is well supported. A controlled ablation shows read and write each help and work best combined, and RoboStereo beats the one-way WAFT-Stereo to Pi3X pipeline on all groups.
Critical weaknesses
  • Data and architecture are confounded.
    • Baselines were not trained on the benchmark groups.
    • The largest margins (77 to 81%) come on groups seen in training, while held-out gains range from about 0.5% to 42%.
    • No stereo baseline is retrained on the same data.
  • Synthetic-only quantitative evaluation. Real-world results are only qualitative, so the claimed benefit for real embodied interaction is not established.
Questions for authors

How does WAFT-Stereo fine-tuned on the same training mixture compare? Can you provide any quantitative real-world evaluation with depth ground truth? What are runtime and memory compared with DVLT and WAFT-Stereo?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.