acceptodds
Withdrawn from ICLR 2027

ContextDepth: Boosting Metric Depth Estimation via Richer Context

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Monocular metric depth estimation is inherently ill-posed: a single image provides only a keyhole view of the scene, causing models to misestimate absolute scale despite recovering sharp relative geometry. While in-context learning (ICL) steers models through inputs alone, existing visual ICL frameworks focus on instructing generalist models on *which* task to perform. We ask a different question: can context improve *how well* a fixed specialist model executes its task? We propose **ContextDepth**, a training-free spatial ICL framework that improves pretrained metric depth estimators without updating model weights or modifying raw input pixels. ContextDepth surrounds the query with diverse outpainted contexts to induce alternative scale hypotheses, then uses the directional bias from a retrieved depth reference to select the best candidate, leaving predicted depths unmodified. Across six mainstream depth models and six benchmarks, ContextDepth delivers consistent improvements with and without camera intrinsics, boosting δ₁ accuracy by up to 22.65 percentage points on the strongest baseline, Depth Anything 3. Crucially, guided selection remains robust under cross-dataset and out-of-domain reference banks where direct rescaling collapses, while in-domain banks yield the largest gains. These results establish visual context as an effective lever, unlocking substantial headroom as generative priors and reference banks scale.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Metric Depth Estimation, In-Context Learning, Image Outpainting, Monocular Depth Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_d3uMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ContextDepth. During the inference of one image, ContextDepth first generates 3 outpainted images using OneReward and runs a off-the-shelf depth model on each. ContextDepth then selects a candidate based on the depth model's error on an retrived RGB - depth sample from an external bank (by default, mixing all 6 datasets). Experiments cover 6 models and 6 datasets but mainly on centrally cropped images.

Metrics depth models' performances are commonly much worse on boundaries than central areas. But this paper is trying to hiding this. The outpainting of 1 single case contributes much more than selecting from multiple candidates. The most significant gains are on centrally cropped images but not on full images. And not all depth models yield improved performance when this method is applied. I lean towards clear rejection of this paper.

Critical strengths
  1. Experimental results show significant gains. In table 1, DA3 without intrinsics was improved from 51.17% to 68.47% in δ1\delta_1 with one outpainting and 73.82% with selection.

  2. In figure 3, outpainting beats all other padding controls, which supports the importance of outpainting meaningful content.

  3. Table 2 compares default mixed bank, cross-dataset bank, and single-dataset bank, which clearly tells the effect when retriving from different references.

Critical weaknesses
  1. The main results, which supports the "significant gains", is conducted on images that are centered cropped to 2/3 along each dimension. However, in full-image experiments, 3 of 6 models deliver worse results when the intrinsics are unknown. Yes, all 6 improve when intrinsics are known in full-image experiments, but the improvements are little. Whether such improvements are worthwhile under the huge additional computational costs and much slower speed is questionable. 3 outpainting using the full multi-step denoising, the latency of a single denoising step can be enough for some depth mdoels to predict depth for an image, let along the retriving, multiple depth model inferencing, and selecting. These significnatly limits the evidence for the claim of this paper. Though ignoring these limitations, figure 1's "resolving scale ambiguity" should also be narrowed down to "reducing" not "resolving".

  2. The selection's contribution also needs more clarification. In table 1's without intrinsics branch, 1 outpainting contributs much more than selecting from 3. However, the full method includes 3 outpaintings, 3 retrivings, and 3 + 3 (3 outpainted images and 3 retrived images) depth model infereces. The latencies of each stage and the entire end-to-end pipeline are missing. The sweep of the number of candidates is also missing. Table 3 rescales "Query-only" predictions, however, the proposed method selects among predictions on outpainted images, and therefore does not cleanly isolate the benefit of selection over rescaling.

Questions for authors

see weaknesses

Reviewer_YbbXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ContextDepth proposes a training-free framework for improving frozen monocular metric-depth estimators through generated spatial context. It outpaints several surroundings while preserving the original query pixels, then uses a retrieved reference’s ground-truth depth and camera intrinsics to select a candidate prediction without rescaling it. Evaluation covers six models and six datasets, primarily using cropped queries. The complete method improves both reported metrics in all twelve aggregate model–camera settings. Padding controls, reference-bank comparisons, and selector ablations investigate the contributions of generated content and reference guidance.

Critical strengths
  • The paper provide a broad evaluation with useful component controls. The experiments span independently developed estimators, known and unknown query intrinsics, and several reference-bank configurations.

  • The paper writing is clear but it is a bit long in my opinion.

Critical weaknesses
  • The complete method depends on annotated references at inference, and the benefit over simpler use of that information remains insufficiently established. Reference-guided selection requires RGB images with ground-truth depth and known intrinsics. Reference depth remains necessary even when query intrinsics are known. Consequently, the complete method requires access to external metric annotations at deployment. The cross-dataset and fixed-DIODE experiments demonstrate robustness to bank composition, but do not address bank absence. The bank-free “Outpaint” variant is evaluated and provides substantial gains; however, there is no explicit missing-bank deployment policy. Furthermore, a query-only baseline supplied with the retrieved reference’s appropriately normalized focal lengths would help determine how much of the unknown-intrinsics improvement requires context generation rather than straightforward use of the available calibration.

  • The demonstrated improvement is substantially less general on full-image inputs. The principal experiments remove existing context by taking center crops spanning approximately two-thirds of the original width and height. In the full-image setting without intrinsics, the complete method reduces δ₁ from 80.86 to 72.52 for UniK3D, 62.09 to 49.85 for Depth Pro, and 69.48 to 66.32 for UniDepth V2, with AbsRel also worsening (Table 10). The authors acknowledge these failures, and the remaining models improve. Nevertheless, substantial degradation for half the evaluated models limits practical applicability, particularly without a criterion for determining when context expansion should be used.

Questions for authors
  • The paper uses outpainting methods to generate novel regions. Let say the original input image has a table and the outpainting method generate two candidates that one is an indoor and one is outdoor images. How does this method handle such cases where the scale metric depth of candidate images are so different ?

  • When no annotated reference bank is available, is fixed single-candidate Outpaint the recommended configuration? Given the full-image degradations in Table 10, how should users determine whether to apply outpainting or retain Query-only inference?

  • Does pasting the original ROI back into the generated canvas introduce discontinuities in appearance, contours, or perspective at its boundary? Pixelwise equality verifies preservation of the query, but does not establish consistency with its surroundings. Please examine whether errors concentrate near the boundary and whether excluding a narrow boundary band materially changes the gains. This would clarify whether restoration artifacts affect local geometry or global scale.

  • How does query-only inference perform when supplied with the retrieved reference’s normalized focal lengths, appropriately mapped to the query coordinates and using identical references? This comparison could materially change the interpretation of the largest unknown-intrinsics gains.

Reviewer_z6mUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper suggests that monocular metric depth estimation models err at estimating absolute scale because a single image offers a limited view of the scene. It proposes a training-free method that adds generated context around the image (outpainting) to improve the predictions of frozen depth models. Its contributions rely on two main components: (i) outpainting the input image into several candidate contexts and running the frozen model on each; and (ii) selecting one of the candidates using a retrieved reference image with known depth. The method is tested across several off-the-shelf depth estimation models.

Critical strengths
  • The proposed method is simple, training-free, and works across any off-the-shelf monocular depth estimation models (the method is implemented across six different backbones).
  • The paper is very clearly presented. It is easy to follow and well-organized. Extensive experiments are provided, including candid ablations and failure cases.
Critical weaknesses

The main concern is that the experimental settings of the main results are designed in favor of the proposed method, contrasting with the baselines’ natural setting.

  • For instance, the baselines are evaluated on centered 2/3 crops of the original images (Tab. 1-3), which is not a fair setting. Given that the center crop keeps the same camera position and focal length in pixels but makes the field of view more narrow, depth models which tend to assume a typical field of view may then underestimate the focal length and consequently the metric depth (e.g., note the very noticeable drop of DA3 in δ1 on cropped KITTI without intrinsics compared to with intrinsics, Tab. 7). This is further supported by how three out of the six models perform worse with ContextDepth when the full uncropped image is provided as input (without intrinsics, Tab. 10). In this scenario, widening a normal photo is expected to make models overestimate the focal length (it would be useful for authors to show the direction of the error).
  • Comparing the results from Tables 1 and 10, ContextDepth results on the cropped setting remain close to the baselines on the full images without intrinsics (within ~4.5 pp δ1, three models above and three models below), and below them with intrinsics in five of the six models. Since the two tables are evaluated on slightly different regions - cropped vs. full canvas -, it would be useful for the authors to report the full-image baselines from Tab. 10 on the same central region of Tab. 1 for a direct comparison. Moreover, within Table 10, it is also observable how, when intrinsics are provided, ContextDepth improves only modestly, and a single outpainting alone is worse than the baseline for half the models. This suggests that the remaining gain seems to come mainly from the selection step (see next weakness).
  • As a result, the main reported gains seem to stem from the recovery from a distortion introduced by the evaluation protocol, and not from the general claim that generated context improves scale estimation. The paper does not justify the use of the cropping setting used in the main results (nothing in the method requires it), and the full-image setting in App. §D.4 should be the main setting, with per-dataset results.

Another concern is how the proposed reference-guided selection step relies on the assumption that depth models err in the same direction on similar scenes (§3.2.1), however, no empirical evidence is provided to support this. It would be useful for the authors to provide how often the sign of the reference error matches the one from the query. Moreover, the ablation in Tab. 12 suggests that the selection gain comes mostly from matching the focal length of the reference, and Tab. 11 reveals that it largely disappears when the reference comes from a different dataset. As images from the same dataset are typically captured with the same or similar cameras, the selection may mainly benefit from the camera information of in-domain references, rather than from the proposed directional bias (it would be useful for the authors to report retrieved reference intrinsics vs. the query ones).

Questions for authors

Despite the mentioned strengths, the main reported gains seem to result from the cropped evaluation setting (W1), and the selection step appears to rely mainly on in-domain camera information (W2). I believe the idea has potential, but it would require major revisions. In my view, the full-image setting should be the main evaluation, and the gains there should be more convincing. A clearer analysis of the selection step would also be important. The following questions would provide more clarity:

  • Could the authors compare the focal length estimates of the depth models on the center crop queries, outpainted candidates, original full image, against ground truth?
  • Could the authors report the full-image results of Tab. 10 per dataset? Additionally, on the same central crop region as Tab. 1?
  • How are known intrinsics adjusted for the crop and expanded canvas?
  • Would the method improve performance on in-the-wild images with a genuine narrow field of view (e.g., telephoto)? As opposed to center crops with in-domain references

Comments on the paper

2 comments

We thank all three reviewers for their careful evaluation and constructive feedback. We have added experiments, updated results, and clarified the method and protocol. We first address shared concerns.

For concerns about cropped inputs and full-image results, we added real-context and camera-controlled experiments to test how existing context affects gains and whether camera re-estimation after expansion causes degradation. We updated Tables 1 and 10 and examined results from a single outpainting to clarify whether gains depend on cropping and when the method is applicable.

The real-context experiments cover 3,406 images from six datasets, DIODE, ETH3D, HAMMER, KITTI, NYUv2, and iBims-1. Starting from a central crop of approximately two-thirds of the original width and height, we restore 25%, 50%, 75%, and 100% of the removed periphery, producing five nested inputs ending with the full image. All added pixels come from the same original image, with original sampling and no extra resizing, generated content, or selector. We test all six models with and without GT intrinsics, giving 12 model and camera combinations.

For each of the four adjacent input pairs, we infer depth from both inputs and crop the larger prediction back to the smaller input's region. We compare δ1 and AbsRel on exactly the same valid GT pixels, with equal weight per image, to measure the benefit of added surroundings. With GT intrinsics, calibration always accounts for cropping, canvas expansion, and internal resizing. Without GT intrinsics, both Uni models reuse the smaller query's estimated camera, mapped to the larger canvas; the other four retain native camera recovery.

Figures 9 and 10 on pages 26 and 27 of the revised manuscript show that, with GT intrinsics, all five models except Depth Pro improve both metrics on average at every expansion. UniDepth V2, DA3, and UniK3D generally show diminishing gains; the MoGe models maintain more consistent positive gains. Depth Pro's gains fluctuate near zero even with GT intrinsics, indicating little consistent benefit from extra context. Without GT intrinsics, it independently predicts focal length and canonical inverse depth, then converts to metric depth using the focal-length/image-width ratio. Focal-length errors propagate directly into global scale errors. As expansion changes these predictions, camera-induced scale changes can outweigh the limited contextual gains. Limited context response and sensitivity to camera errors together explain its weaker performance under ContextDepth; similar degradation occurs with real context. The Uni models instead feed camera rays into depth decoding; MoGe and DA3 recover scale through shared scene representations and geometric or metric branches. Camera errors thus follow different paths. See Appendix G.2 on page 27, and G.5 on page 30 for Depth Pro controls and implementation checks.

To address z6mU's concern about FOV and camera bias, we keep generated content unchanged and control the cameras of both Uni models without GT intrinsics for a fairer comparison. We map the Query-only camera to the expanded canvas to generate decoding rays, replacing the camera re-estimated from the expanded image. We control the Uni models because their depth decoders explicitly use camera estimates; the other four retain native inference. Changing only camera handling turns UniK3D and UniDepth V2's full-image seed-0 δ1 changes from −9.48 and −3.96 in original Table 10 on page 21 to +0.18 and +1.74 in Table 14, respectively. Crop gains remain +11.88 and +7.85. Updated versions of Tables 1 and 10 appear on pages 28 and 29. These two degradations thus arose from adverse camera re-estimation that obscured contextual gains. Controlling it preserves gains on both crops and full images.

Whether smaller full-image gains justify extra computation and latency should be assessed across input conditions: marginal gains are smaller on full images and much larger on central crops. Tight framing or post-capture cropping produces valid RGB inputs, and users cannot judge context sufficiency in advance. For the five models that use context, a single outpainting already gives substantial crop gains and largely preserves or improves full-image accuracy, with only small declines in a few metrics. Reference-guided selection further improves accuracy and stability. We therefore support ContextDepth for these five models in both camera settings, with or without the selector, without requiring per-image context assessment. Depth Pro without GT intrinsics is excluded from this default recommendation for the two reasons above.

I thank the authors for the response and the additional experiments. Regarding my initial review, Q3 is addressed and W1 is only partially addressed, while W2, Q1, Q2, and Q4 are not addressed. In my view, crucial concerns remain:

  • W1: Main results and claims still rely on center crops, which is not a fair setting for the baselines, and the argument that these are a practical input condition is not well motivated. For the models with the largest gains (MoGe-2, MoGe-3, DA3), the camera is still re-estimated on the expanded canvas, so it is not clear whether their gains come from the generated content or from a better camera estimate once the original field of view is restored (Q1). On full images with intrinsics, a single outpainting barely improves over baselines (Tab. 14).
  • W2: The selection step, which seems to account for most of the remaining gains, is still not analyzed, and it may mainly benefit from the camera information of in-domain references.

Therefore, I will keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.