acceptodds
Under review as a conference paper at ICLR 2027

DepthAgent: Advancing Geometry-Adaptive Depth Estimation via Sample-wise Expert Selection

Abstract

Monocular metric depth estimation has achieved strong progress with large-scale training and universal-camera modeling, yet robust deployment across diverse camera settings, such as perspective, fisheye, and panoramic images, remains challenging. Existing methods typically rely on a single depth estimator, overlooking that different models encode different camera assumptions and perform best under different input domains. In this paper, we show that depth experts exhibit strong sample-wise complementarity: model preference is highly correlated with camera geometry, and multi-model fusion brings the largest gains on difficult samples where individual experts are unreliable. Motivated by these observations, we propose , a vision-language agent for geometry-adaptive monocular depth estimation. DepthAgent treats existing depth models as frozen tools and learns to analyze scene and camera cues, invoke suitable experts through multi-turn tool utilization, and select or fuse their predictions for each input. To optimize such discrete decision-making toward dense geometric quality, we design a multi-reward reinforcement fine-tuning scheme that jointly encourages valid tool execution, camera/scene analysis, expert-selection quality, and inference efficiency. Extensive experiments across perspective, fisheye, and panoramic benchmarks show that DepthAgent consistently outperforms individual experts, fixed model fusion, and rule-based and learned routers with fewer expert calls, with strong improvements on challenging samples. The code and model will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Agent, Depth Estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HnCgOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper finds that existing off-the-shelf monocular depth estimation models are not robust across diverse camera settings. More specifically, it finds that a single model only excels in a certain range of camera configurations, and multi-model fusion can outperform single model.

Based on this finding, authors propose DepthAgent, an agentic framework which uses off-the-shelf expert depth models as tools to produce better depth prediction. More specifically, DepthAgent chooses some expert models, and runs inference of those models on the input image, then combines the output of different expert models.

Authors evaluate DepthAgent on benchmarks covering diverse camera configurations, including perspective, fisheye and panoramic, and DepthAgent shows a consistent improvement, compared with both single expert baselines and other fusion-based methods on each camera configuration.

Critical strengths
  • Though it is expected that existing monocular depth estimation models are not robust to diverse camera settings, this paper did thorough quantitative analysis on this. Their analysis cover both perspective-oriented and EPR-trained monocular depth estimation models, and show that multi-model fusion can bring consistent improvement.

  • DepthAgent provides more accurate depth predictions within a reasonably fast inference time. More specifically, DepthAgent outperforms all single expert baselines and other fusion-based methods in all perspective, fisheye and panoramic datasets, and its fast mode is only 0.5x the speed of running all 5 expert models.

  • The methodology is general in the sense that it can be applied to any given set of expert models, and the performance may potentially improve when export models become better.

Critical weaknesses
  • The novelty of building an agent is limited. The main methodology of building an agent used in this paper is: treating frozen expert models as tools, choosing which expert to call, and combining their outputs. This idea is used in [1]. Though the authors mention [1] in related works, there is not enough discussion on how DepthAgent differs from that, besides on a different task.

  • The evidence that no-retrain is needed to the depth agent when depth expert pool changes (Line 252) is not convincing, and the cost of re-training could be a limitation of this method when better expert models are available. In Appendix E, the author only did experiments of removing 1 expert and replacing an expert (UniDepth) with a similar model (UniDepthV2). Authors do not mention how to address cases when new models are added to the expert pool, and substantial changes are made to the expert pool (e.g. multiple models are removed/replaced/added).

  • The writing about how DepthAgent produces its output is not clear. In Sec 3.2, the author mentions DepthAgent does multi-turn execution of expert depth models, but there is little description on how DepthAgent produces output from the output of those expert depth models. Sec 3.1 mentions some available options, but are these options the same as those being used by the DepthAgent? From Fig. 5(b) and 8, it seems that DepthAgent is able to output “avg” of 2 outputs from expert models, but still, there lacks a detailed description on what options are available.

[1] FusionAgent: A Multimodal Agent with Dynamic Model Selection for Human Recognition. Jie Zhu, Xiao Guo, Yiyang Su, Anil Jain, Xiaoming Liu; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 32756-32766

Questions for authors
  • Can authors highlight their novelty compared with prior work, especially FusionAgent, besides addressing a different task?

  • How should the agent adapt when new models are added to the expert pool or multiple models are removed/replaced?

  • Can authors clarify how DepthAgent produces its output based on output from expert models? Especially, what fusion options can DepthAgent perform, and are those options performed over the whole image or can be applied on a selected subregion of an image?

Reviewer_w7dGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose DepthAgent, an agentic framework for depth map fusion from multiple experts. The depth agent receives image/camera information and utilizes monocular depth estimation networks as tools. The agent utilizes the tools to compute depth maps and depth statistics and decides whether more experts need to be called or a fused depth prediction can be formed. The experimental results show strong performance when compared to the individual experts.

Critical strengths

The experimental evaluation is strong and the authors show clear improvement across all metrics when compared to the baseline methods. The agent is able to fuse multiple results instead of converging to the strongest expert, which means that there are enough complementary cues to guide the agent to the correct decision. The authors conduct experiments with different types of cameras and different routing strategies. The paper gives a modern agentic perspective on depth expert fusion.

Critical weaknesses
  1. Although the results show that there is some selection and mixture of the depth estimates, in figure 5. it is evident that the results for fishey and panoramic are dominated by a mixture of UniDAC + UniK3D which makes the claims in the paper weaker than stated. It also raises the question: if there were a better expert, would that expert dominate the result?

  2. As a consequence of 1, depth anything is a very strong monocular estimator but the authors do not include it in the toolbox or compare against it. Results including depth anything comparisons would further strengthen the claims in the paper.

  3. In the paper, it is not stated how deterministic the results are or what the standard deviation is of running the depth agent on the same samples over several runs. Is the expectation that depth agent will always produce the same fused depth and the tool selection is deterministic over multiple runs?

Questions for authors

Look at questions above

Reviewer_s3uKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an agentic framework for monocular depth estimation, where a VLM orchestrates off-the-shelf depth experts via multi-reward RFT, motivated by the claim that expert preference is camera-geometry-dependent. Strengths are real: expert complementarity is characterized systematically (multi-model oracle solutions for 78.6% of samples, gains concentrated on the hardest quintile), and plug-and-play is demonstrated—experts can be removed or replaced at inference without retraining while still beating re-fit baselines. Two critical weaknesses undermine the central claims. First, Table 1 contradicts the causal motivation: changing projection with the dataset fixed barely moves preference (80.1% → 77.1%), whereas changing the dataset with projection fixed flips it dramatically (77.1% → 2.2%). Preference is driven by data source and acquisition domain, not projection geometry—removing the rationale for the geometry-aware design, a discrepancy never discussed. Second, no component is shown necessary: the strongest single expert reaches δ1 = 0.937 versus 0.948 (a 1.1-point margin), no selection distribution is reported (collapse to "always pick the strongest generalist" cannot be ruled out), there is no zero-shot VLM routing baseline without GRPO, no non-agentic confidence-based fusion baseline, and the learned-router baseline is a strawman (MLP 0.904 < naive min-fusion 0.912). It thus remains unclear whether gains come from adaptivity, RL training, or the agentic formulation. Presentation issues compound this: terminology is inconsistent across text, figures, and tables, and the training/inference distinction is unclear. The authors' questions are well-posed and could materially affect the assessment if answered convincingly. As it stands, the paper offers a useful descriptive study and a demonstrated orchestration property, but its causal motivation is contradicted by its own data and its necessity claims lack the required ablations. I lean toward weak rejection.

Critical strengths
  1. The empirical characterization of expert complementarity is systematic, quantitative, and useful to the community. Rather than asserting that model selection matters, the authors measure it: oracle solutions are multi-model for 78.6% of samples, and fusion gains are not uniform but concentrate on the hardest quintile. This is consequential because non-uniform gains are precisely what a fixed policy cannot exploit, so the study supplies a quantitative basis for input-adaptive depth estimation — a topic that remains underexplored. This finding is descriptive and stands independently of the paper's causal story about camera geometry.
  2. The plug-and-play property is demonstrated, not merely claimed. The authors modify the expert pool at inference without retraining: when an expert is removed, the agent stops calling it and still outperforms baselines that are allowed to re-fit; when an expert is replaced by an unseen one, it is adopted from its description alone. This is consequential because it establishes practical value for orchestration-based formulation that does not depend on the causal motivation in W1, and it addresses a real deployment concern — integrating new depth models without retraining.
Critical weaknesses
  1. The paper's own data contradict the claim that expert preference is driven by camera geometry. The contribution list states preference is camera-dependent (correlational), but this is used to motivate a causal claim: that a VLM should infer camera geometry to select experts. Table 1 adjudicates between these. Changing projection format while holding the dataset fixed (Perspective → ERP-variant) barely moves preference (80.1% → 77.1%); holding projection fixed while changing the dataset (ERP-variant → Native-ERP, both ERP) flips it dramatically (77.1% → 2.2%). Preference is thus driven primarily by data source and acquisition domain, not by projection geometry. This is consequential because it removes the stated rationale for the geometry-aware VLM design, and the discrepancy is never discussed.
  2. Every ingredient is off-the-shelf, so the contribution must be established by ablation — and the decisive comparisons are missing. The strongest single expert, UniK3D, already achieves δ1= 0.937 on perspective data versus DepthAgent's 0.948, a margin of only 1.1 points, and no selection distribution is reported, so collapse to "always pick the strongest generalist" cannot be ruled out. There is no zero-shot VLM routing baseline without GRPO, so the contribution of multi-reward RFT relative to the VLM's intrinsic capability is unknown. There is also no non-agentic confidence-based fusion baseline. The learned-router baseline is additionally a strawman (MLP 0.904 < naive min-fusion 0.912). This is consequential because the headline number sits close to the best single model, and the paper does not establish whether the gains come from adaptivity, from RL training, or from the agentic formulation.
Questions for authors
  1. What fraction of samples does the agent resolve with UniK3D alone? Give the per-dataset breakdown of the 0.937 → 0.948 margin.
  2. Zero-shot VLM routing without GRPO — how much does RFT actually add?
  3. Is fisheye in training? Sec. 3.1 contains no fisheye data, yet fisheye results are reported. Given W1, unseen-fisheye generalization is the only remaining support for the geometry story.
  4. Reconcile the figure with Sec. 4; confirm every block maps to a real component and rewards are training-only.
  5. Fast mode is 0.98 s end-to-end vs. 0.51 s for all five experts — "1.9 expert calls" does not imply lower latency. Report wall-clock/FLOPs and the full Pareto curve. Also report RMSE/Abs Rel on hard samples, and discuss min-fusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.