RoboStereo: Recurrent 4D Reconstruction from Stereo Videos for Embodied Interaction

Embodied robots require accurate 3D geometry during close-range interactions, yet object motion and limited relative viewpoint changes can leave local geometry weakly constrained in monocular video reconstruction. Synchronized stereo provides geometric evidence, but textureless regions, repetitive patterns, and occlusion can make matching ambiguous. We present RoboStereo, a recurrent model that reconstructs per-frame dense geometry and cameras in a shared coordinate system from synchronized, rectified stereo videos. Within a single inference run, multi-frame representations repeatedly guide stereo matching, while matching features feed back into the reconstruction state. To address the limited scene diversity of existing stereo data, we render annotated stereo videos for training and evaluation and construct the Embodiment Stereo Benchmark from selected renderings, with an emphasis on dynamic scenes and embodied manipulation. Experiments on our benchmark and public datasets demonstrate improved scale-aligned depth estimation over stereo and monocular multi-frame baselines, including on manipulation datasets excluded from joint training. Qualitative comparisons further show clearer local geometry around hand–object interactions.