acceptodds
Withdrawn from ICLR 2027

EGODEPTH: A RESIDUAL-BASED STEREO VIDEO DEPTH PREDICTION METHOD WITH 4D SPATIOTEM￾PORAL FEATURE FUSION

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Accurate and temporally consistent stereo matching is essential for robotic percep- tion. Frame-wise stereo foundation models provide accurate disparity estimates but lack explicit temporal modeling. We present EgoDepth, a method for tempo- rally consistent stereo matching built on Stereo Any Video. Disparity estimates from a frozen FoundationStereo model initialize single-scale recurrent refinement. During training, confidence-weighted residual regularization constrains deviations from the prior, with confidence derived from left-right photometric consistency. With its trainable components optimized on SceneFlow, EgoDepth achieves the lowest end-point error (EPE) on three of four cross-domain benchmarks and the lowest temporal end-point error (TEPE) on all four among the evaluated methods. After mixed-data training, EgoDepth also outperforms Stereo Any Video in both EPE and TEPE on Sintel Final and KITTI Depth. Real-world evaluations fur- ther support the generalization of EgoDepth across different scenes and capture settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Stereo matching, Video depth estimation, Temporal consistency, Foundation models, Cross-domain generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MhjROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents EgoDepth, a framework for stereo video depth estimation.

EgoDepth builds on two existing frameworks, Stereo Video Anything (SVA) and FoundationStereo, and combines them. Specifically, it uses FoundationStereo disparity predictions as initialization for SVA iterative inference. Additionally, EgoDepth adds a few minor modifications to SVA, such as operating 3D GRU refinement at a single scale rather than at multiple scales, and introduces a regularization term that encourages EgoDepth predictions not to deviate excessively from the initial FoundationStereo predictions. A fast variant is proposed alongside EgoDepth, reducing inference time by roughly 2×2\times.

Experimental results on standard stereo video datasets such as Sintel, Dynamic Replica, Infinigen Stereo, vKITTI, and KITTI establish EgoDepth as state-of-the-art in video stereo depth estimation, while ablation studies conducted on FlyingThings3D measure the impact of the different design choices. A final experiment is reported, comparing the accuracy of EgoDepth against a commercial IR-based depth sensor on images containing calibration patterns.

Critical strengths

The main strength of this paper definitely is in the quantitative results.

  • Tables 1 and 2 highlight that EgoDepth achieves state-of-the-art stereo video depth estimation.

  • Table 3 studies the impact of the different components and design choices, showing that FoundationStereo and single-scale 3D GRU processing play the main role in the whole framework.

Critical weaknesses

The paper, unfortunately, exposes several critical weaknesses:

  • Limited novelty and technical contribution. The method itself is mainly a simple combination of existing components: Stereo Any Video (SVA), the previous state of the art in stereo video depth estimation, and FoundationStereo. Other design choices departing from SVA consist of using single-scale 3D GRU processing rather than multi-scale processing (whose effectiveness is known: the original RAFT paper used a single-scale approach), plus a regularization term during training to encourage final predictions to be consistent with FoundationStereo ones, whose impact is the least significant. This amounts to a limited contribution.

  • Undefined fast variant. The text refers to a fast variant of the method, but it is never formally introduced or described anywhere in the paper. Accordingly, it is hard to consider EgoDepth-fast a contribution itself, in particular if it only differs from the original EgoDepth by using mixed-precision, TensorRT compilation, and Triton-based operation fusion rather than in terms of topological design.

  • Unbalanced paper structure. The method section covers little more than a single page, while two pages go to an "Application" section reporting experiments of questionable value. Indeed, adding a comparison between an IR-based depth sensor and EgoDepth alone adds little value to the paper, in particular in the absence of any other stereo backbone reported in this experiment. This reinforces the impression of a thin contribution. In addition, the related work section is barely half a page, looking more like a list of references without any explanation of what the different methods propose. This would be acceptable in a very dense manuscript, which seems not to be the case here.

  • Insufficient experimental evaluation. The lack of novelty alone could be compensated for in case of thorough comparisons and studies, which are currently absent. For instance, FoundationStereo is used as a backbone without any empirical comparison with other backbones (e.g., Selective-IGEV, MonSter, Stereo Anywhere, or DEFOM-Stereo).

Questions for authors

To address the aforementioned critical weaknesses, I would ask the authors to provide the following during the rebuttal:

  • Sensitivity study across stereo backbones. Please report results when initializing the method with different stereo backbones, such as Iterative-IGEV, MonSter, Stereo Anywhere, and DEFOM-Stereo. This would show whether the gains hold beyond the specific FoundationStereo + SVA combination and would make the evaluation considerably more exhaustive.

  • Formal introduction to the fast variant. Please describe EgoDepth-fast in detail (architecture, differences from the full model) and report its accuracy and runtime. At the current stage, it is hard to consider it a contribution by itself.

  • Revise the "Application" section. The current content adds very little to the paper. Replacing the current experiments with some other kind of evaluation on a higher-level application would definitely add more value to the manuscript. For instance, showing that the more accurate disparity maps can improve a downstream task, such as surface reconstruction with a framework like GS2Mesh (Wolf et al., 2022), would demonstrate practical value far better than the current experiments.

Depending on the answers provided to these points, I could consider increasing my score despite the rather limited technical novelty.

Reviewer_aMvNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EgoDepth combines frozen FoundationStereo predictions with the temporal refinement architecture of Stereo Any Video (SAV). It replaces multi-scale refinement with a single-scale recurrent module and adds a photometric-confidence-weighted penalty that anchors predictions to the foundation prior. Experiments report improvements in disparity accuracy and temporal consistency, together with an accelerated implementation and real-world depth measurements.

Critical strengths
  1. The paper addresses the useful problem of improving temporal consistency without losing the accuracy of a strong frame-wise stereo model.

  2. Evaluation covers several stereo-video benchmarks and reports both spatial and temporal errors.

  3. Table 3 separates foundation initialization, refinement scale, and residual regularization rather than reporting only the full model.

  4. The calibration-target experiment provides quantitative real-world evidence beyond qualitative depth maps.

Critical weaknesses
  1. The gains over the foundation prior and the contribution of the new regularizer are insufficiently isolated. FoundationStereo alone is missing from Table 1. Table 3 shows that initialization and single-scale refinement account for most of the improvement; residual regularization changes EPE from 0.576 to 0.555 and has essentially no benefit in the multi-scale setting. These results support the overall combination, but do not yet establish the value of confidence weighting or its role in cross-domain generalization. The key controls are the same foundation checkpoint alone, refinement without temporal interaction, and uniform versus confidence-weighted regularization, evaluated on both EPE and TEPE.
  2. The photometric confidence requires stronger justification. Eq. (5) takes the smaller reconstruction error over both positive and negative horizontal disparity shifts at each pixel. For a rectified pair with a fixed disparity convention, only one direction represents the intended correspondence. The alternative shift can produce an accidental match and inflate confidence, particularly on repetitive or weakly textured regions. The treatment of out-of-bounds samples and occlusions is also unclear. Since these weights determine how strongly predictions are anchored to the teacher, their relationship with actual teacher error should be measured.
Questions for authors

How much do temporal refinement and confidence weighting improve over the identical foundation prior, particularly on genuinely unseen domains?

Why is the minimum over both warp directions used in Eq. (5), and how well does this confidence predict disparity error?

Does the runtime advantage remain when SAV uses the same precision and compilation backend?

Reviewer_iTmmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes EgoDepth, a stereo video depth method that combines FoundationStereo with Stereo Any Video (SAV). The idea is to start from a strong per-frame depth prediction model and use temporal refinement to improve consistency across frames. The method simplifies SAV to a single refinement scale and adds a confidence-weighted loss that discourages unnecessary changes to the initial prediction. The experiments report improved depth accuracy and temporal consistency, along with a faster implementation and some real-world examples.

Critical strengths
  • The general idea makes sense and seems to be convincing. A strong stereo model should already recover most of the scene geometry, so the video model can focus on temporal corrections instead of estimating depth from scratch. I also like that the authors simplify the temporal model rather than simply making it larger by adding more capacity.

  • It also seems that this combination of design choices being made in this paper helps improving the metrics. In Table 2, EgoDepth improves both frame-wise and temporal error over FoundationStereo on UnrealStereo4K, and improves temporal consistency on KITTI while keeping the same frame-wise error. The faster version also retains similar accuracy.

Critical weaknesses
  • My main concern is that, most of the machinery comes from FoundationStereo and SAV, so the additional design choices are single-scale refinement and a prior-preserving loss. This is a reasonable recipe, but I am not yet convinced that the particular temporal model is needed, rather than a simpler temporal correction on top of FoundationStereo. Table 2 shows that refinement helps, so the question for me is whether that benefit justifies running both models and training this pipeline. I would find a comparison against a lightweight temporal refinement of the same stereo predictions more informative than comparisons mainly against weaker image models or the original SAV.

  • I also felt that the paper does not make a strong case for the confidence-weighted regularizer as a main contribution. Table 3 suggests that most of the improvement comes from the strong initialization with a much smaller additional gain from regularization. A simple unweighted prior loss would help establish whether the confidence mechanism is doing something important. More generally, when should the model trust the prior, and when can the temporal evidence correct a prior that is confidently wrong?

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.