acceptodds
Under review as a conference paper at ICLR 2027

From Relative to Metric: Grounding Diffusion Depth with Scene-Structured Language

Abstract

Monocular relative depth models generalize well across diverse scenes, but their predictions are defined only up to an unknown metric gauge. Recovering physical depth therefore requires not only reliable scene geometry, but also a mechanism for grounding that geometry in metric scale. We investigate whether the semantic priors already encoded in diffusion-based depth models can support such grounding without camera metadata, external geometric measurements, or test-time ground-truth alignment. Our key observation is that scale-relevant semantic cues are not geometrically interchangeable: foreground objects, midground structures, and background perspective provide complementary evidence that plays different spatial roles in metric calibration. Based on this insight, we propose a hierarchical scene-language-guided calibration framework for frozen diffusion-depth models. The framework reuses both the relative disparity and intermediate spatial features of the frozen backbone, and introduces depth-structured descriptions for foreground, midground, and background regions to guide spatially varying scale–shift prediction through shared cross-attention. We further propose Scene-Scale Anchor Alignment (SSAA), which complements local calibration by organizing the learned representation according to scene-level metric scale. Experiments with Marigold, Lotus, and E2E-FT across multiple metric-depth benchmarks show consistent improvements and favorable performance against existing relative-to-metric methods with a lightweight trainable module.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Monocular depth estimation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QKuQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a metric depth calibration model for frozen diffusion-baseed relative depth models. It first runs the frozen diffusion-baseed relative depth models for their intermediate features. Then, it predicts the scale and shift conditioned on foreground / midground / background textual descriptions. It is applied on 4 diffusion-based depth models and are trained on 3 datasets using Scene-Scale Anchor Alignment (SSAA) loss based on log-median depth and KL divergence.

Critical strengths
  1. Using the intermediate feature is practical, the intermediate feature of frozen diffusion-baseed relative depth models already captures both RGB image fetures and depth features. So we can keep the calibration model light and small.

  2. Table 6 shows consistently improved AbsRel and δ1\delta_1 when more designs are applied.

  3. Rich supplementary materials. Supplementary's table 7 compares local descriptions with repeating the global description; table 10 compares different grouping methods; and table 9 compared SSAA with other supervisions.

Critical weaknesses
  1. The calibration method proposed here largely overlapped with TR2M. TR2M already uses additional text conditions for predicting the scale and shift factors that lift a relative depth to metric. In my opinion, the differences are: a. using the intermediate features of frozen diffusion-baseed relative depth models; b. grouping text with depth; and c. the SSAA loss. But the overall method is not that new. Although the experiments cleanly tell that those differences are benefitial, the performance gains of those new designs are modest. I think the paper needs to be re-framed around those differences rather than being framed as a broader method for text-based relative-to-metric depth calibration.

  2. The improvement over TR2M is not consistent on all datasets used for evaluation. In main paper's table 3 and 4, the calibration method proposed in this paper is better on 5 datasets but is worse on 2. And more discussions about this is better added.

Questions for authors

Please see weaknesses.

3 comments

1、Overlap with TR2M and contribution

We agree that the paper should focus on the specific differences you identified. We have revised the abstract, introduction, and conclusion accordingly, explicitly acknowledging that text-guided, pixel-wise relative-to-metric calibration is shared with TR2M. We do not claim the inverse-depth scale–shift operator as new. The revised contribution comprises three design choices:

  1. the calibrator uses intermediate spatial features from the frozen diffusion-depth model, avoiding a separate image encoder for calibration.
  2. foreground/midground/background descriptions condition calibration through shared cross-attention and layer-type embeddings.
  3. SSAA provides an additional training objective based on soft targets derived from each image's ground-truth log-median depth. We agree that the individual gains are modest. Their cumulative zero-shot effect is clearer: Table 6 shows AbsRel decreasing from 0.313 to 0.288 (8.0% relative) and RMSE from 4.189 to 3.951 (5.7%), with δ₁ increasing by 2.6 percentage points over our visual-only calibrator. Adding local descriptions with their fusion and then SSAA yields sequential AbsRel reductions of 3.9% and 3.0%, respectively. These are conditional increments within our framework, rather than matched-condition gains over TR2M or independent evidence that effect size establishes novelty. We therefore frame the work around these specific designs and their empirical evaluation within the existing calibration paradigm, with the scope of each comparison stated explicitly.

2、Non-uniform improvement across domains

The improvements are not uniform. Across the eight zero-shot datasets in Tables 3 and 4, our method improves both reported metrics over TR2M on five datasets and underperforms on three. Specifically, AbsRel increases from 0.2955 to 0.3204 on DIODE Indoor (+8.43%), from 0.1128 to 0.1379 on DDAD (+22.25%), and from 0.5046 to 0.5854 on DIODE Outdoor (+16.01%). These regressions limit any claim of consistent superiority across domains.

Appendix D.10 now provides a diagnostic comparison of both models on the complete test sets, totaling 19,781 images, with identical masks, crops, and depth ranges. For each image, let e denote the pixel-wise log-depth error, b its mean over valid pixels, and r² its de-meaned variance. The mean squared log-depth error decomposes as mean(e²)=b²+r². We compare dataset means, with Δ defined as our result minus TR2M's. The two DIODE subsets show predominantly scale-related excess error: Δb² is +0.099 on DIODE Indoor and +0.658 on DIODE Outdoor, while Δr² is approximately zero. The DIODE gaps also reverse after test-GT-assisted per-image scale normalization in the offline diagnostic. This supports the importance of image-level scale mismatch in these comparisons. DDAD exhibits a different pattern: the excess error is mainly in the de-meaned residual (Δr²=+0.0065), whereas the scale component is comparable and slightly favors our model (Δb²≈0). Thus, a global scale explanation alone does not account for all three regressions. These diagnostics identify different error compositions, not the causal failure of a particular module. In particular, the residual term should not be equated with a failure of the depth backbone, and GT-assisted scale normalization does not replace the direct metric-depth results or imply improved deployment performance. We have therefore limited the generalization claim to domain-dependent improvements and expanded the discussion of the three underperforming domains. The aggregate ranking is presented alongside the per-dataset results, rather than as evidence of uniformly better transfer.

Dear Reviewer,

We sincerely apologize for disturbing you. Last week we provided point-by-point responses to both of your comments.

In particular, we have re-framed the contribution around the three specific differences you identified — native diffusion-feature reuse, depth-structured language fusion, and soft-anchor scene-scale supervision — and revised the abstract, introduction and conclusion to state explicitly that pixel-wise text-guided relative-to-metric calibration is shared with TR2M. For the non-uniform improvement across domains, we now report the complete per-dataset comparison (five datasets improved, three regressed, with the corresponding AbsRel changes) and have added a diagnostic on the complete test sets, 19,781 images, that decomposes the per-image log-depth error into an image-level scale term and a de-meaned structure term: the two DIODE subsets are scale-dominated, whereas DDAD is structure-dominated. We have narrowed the generalization claim accordingly, and the diagnostic is reported in Appendix D.10.

We would greatly appreciate your feedback on whether these responses adequately address your concerns, or whether any questions remain. As the discussion deadline is approaching, we would be grateful if you could let us know as early as possible, so that we have sufficient time to address any remaining point before the discussion period ends.

Thank you again for your valuable and constructive feedback, which has helped us frame the contribution more precisely and strengthen the empirical evidence.

Best regards,

The Authors

AuthorsOpenReview ↗

Follow-up on contribution clarification and cross-domain results

Dear Reviewer,

As the discussion period is nearing its close, we would appreciate your feedback on whether our rebuttal addresses your two main concerns.

Following your suggestion, we have narrowed the contribution to native diffusion-feature reuse, depth-structured language fusion, and SSAA, while explicitly acknowledging the pixel-wise text-guided calibration formulation shared with TR2M. We also acknowledge the non-uniform transfer results: improvements on five of the eight datasets and regressions on three. The additional scale/structure error analysis examines these regressions and supports a more qualified interpretation of the cross-domain results.

Could you please let us know whether this reframing and the additional analysis resolve your concerns, or whether any point still needs clarification? If your main concerns have been satisfactorily addressed, we would be grateful if you could reconsider your assessment and score in light of the revised claims and additional evidence.

Thank you again for your time and consideration.

Reviewer_1TRqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a metric-depth estimation method that uses a VLM feature. Given a frozen diffusion-based relative method, the paper uses a VLM to obtain a scene description. The scene description goes into a trainable module to output scale and shift maps. These scale and shift maps are utilized to convert the relative depth map from the frozen depth model into a metric-scale depth map (Fig. 2). The method achieves competitive metric depth accuracy compared to competitors.

Overall, the method shows good accuracy on benchmarks, but it lacks justification on design choices and apples-to-apples comparison with direct competitors that also have similar design choices. The ablation study only shows marginal improvement, which questions the validity of the proposed method.

Critical strengths
  • Language-feature guided metric scale recovery

    The paper introduces a lightweight trainable module to recover the scale and shift of relative depth.

  • Competitive accuracy

    On various public benchmark datasets, the proposed method achieves competitive/better accuracy than direct competitors (From Table 1 to Table 4). The ablation study in Table 5, 6, and 7 support the design choices of the method.

Critical weaknesses
  • Lack of insight and justification for outperforming previous methods

    • The paper demonstrates good empirical strengths over direct competitors, such as RSA, TR2M, etc. However, these direct competitors also use a similar pipeline that uses language features to recover scale and shift information. What makes the proposed method outperform the previous method?, and what's the technical novelty of the proposed method over them? Can any apples-to-apples comparison (eg, using the same data, architecture. only techniques differ) validate the design choice?

    • It is difficult to find what proposed design choices make the method stronger than the others. The ablation study table does not fully explain. The proposed designs only give a marginal accuracy boost. This questions the fundamental contributions/novelty of the proposed method.

    • Is it possible to do an apples-to-apples ablation and comparison, starting from the previous method, introducing new elements, and reporting the accuracy improvement?

  • Inconsistent compatibility/accuracy improvement on Marigold

    • In Table 1 and 2, the accuracy actually drops on Marigold: Marigold vs Marigold+our. Can I ask why this happens?
    • Also in Table 5, the conclusion on different backbone architectures is inconsistent. Plus, the numerical gains are so marginal that it is difficult to draw a conclusion if the proposed idea is valid and effective.
  • Table 6: Question on qualitative improvement

    The gains on many metrics are marginal (eg, less than 5%), though it consistently improves the metrics a little. It will be curious to see qualitative improvement from each introduced element, eg, if there is any interesting behavior happening that the metrics fully don't capture.

  • Possible unfair description

    In Table 1, it includes Trainable Parameters for each method. The proposed method includes 23.4M parameters. However, this excludes the total number of parameters that the model uses, such as the pretrained diffusion depth model and the VLM. It will be fair to additionally include the total number of paramters that the model uses during inference.


Due to the concerns on contributions and results stated above, weaknesses outweigh strengths. I rate the paper as Weak Reject.

Questions for authors
  • Unclear details

    • In Equation (3), what is Hook$_{16}$?
    • L190: Why Both RiR_i and Fi16F^{16}_{i} are detached? Can the paper provide any intuition behind this design choice?
    • What is Rescale maps in Table 1?
6 comments

We thank reviewer for the careful reading and the specific questions, all of which we address below.

1、Technical novelty

RSA predicts an image-wide scale–shift pair; TR2M predicts pixel-wise maps. We share TR2M's inverse-depth operator. Our changes are native diffusion-feature reuse, depth-structured text fusion, and SSAA. Fusion uses three weight-sharing cross-attention branches, layer embeddings and a zero-initialized residual projection. SSAA uses fixed scale centres and learned feature anchors. The VLM generates descriptions; calibration uses frozen text-encoder representations, not VLM hidden features.

2、Magnitude of the gains

In Table 6, the full method reduces average zero-shot AbsRel from 0.313 to 0.288 (8.0%) and RMSE from 4.189 to 3.951 (5.7%), with δ₁ up 2.6 percentage points over our visual-only calibrator. Local descriptions/fusion and SSAA give sequential AbsRel reductions of 3.9% and 3.0%. These conditional increments complement the TR2M-based comparison below.

3、Incremental comparison with TR2M

We reconstructed training around TR2M's released model, encoders and base losses. The official weights load with all parameters matched, checking checkpoint compatibility; this does not establish reproduction of the original training recipe or scores. All variants use the same training data and retain all original losses.

Stages

  • B: a single-global-description baseline.
  • L: B plus three local descriptions and layered fusion.
  • S: L plus SSAA.

δ₁ (B / L / S)

  • NYUv2: 0.9280 / 0.9313 / 0.9357
  • KITTI: 0.9480 / 0.9512 / 0.9544
  • SUN RGB-D: 0.5982 / 0.6154 / 0.6359
  • Virtual KITTI 2: 0.8121 / 0.8383 / 0.8409
  • DIML Outdoor: 0.4008 / 0.4230 / 0.4551

AbsRel (B / L / S)

  • SUN RGB-D: 0.3785 / 0.3651 / 0.3561
  • Virtual KITTI 2: 0.1292 / 0.1130 / 0.1156
  • DIML Outdoor: 0.4110 / 0.3845 / 0.3555

δ₁ improves at both steps on all five datasets, although SSAA slightly worsens Virtual KITTI 2 AbsRel. These results support transfer of the additions to a controlled TR2M adaptation. B→L measures the combined content/fusion increment, not organization alone; L→S tests the additional SSAA objective with the original losses retained.

4、Marigold comparison

Marigold Linear Fit fits a per-image scale and shift using test GT; Marigold + Ours predicts metric depth without this alignment. AbsRel is 0.057/0.086 on NYUv2 and 0.088/0.067 on KITTI. These compare a GT-assisted reference with direct metric prediction, so they cannot isolate calibration-induced degradation.

5、Interpretation of Table 5

Table 5 evaluates global text conditioning after per-image GT affine alignment, not final metric calibration. The four models share the SD v2 foundation. Marigold/KITTI δ₁ decreases from 0.8978 to 0.8914, while AbsRel changes from 0.1092 to 0.1095. These small, mixed changes provide a limited compatibility diagnostic, not evidence of consistent improvement or unchanged geometry.

6、Component-wise visualization

Appendix D.11, Fig. 8 compares all four Table 6 configurations on the same five zero-shot scenes, with shared colour scales and per-image AbsRel. Local text shows spatially varying corrections, while SSAA produces changes consistent with reduced scene-scale mismatch. These are observations in the displayed examples, not proof of exclusive component roles.

7 、Complete model size and cost

Appendix D.1 reports 23.376M trainable and 1,314.975M total parameters for prediction with cached descriptions. Rounded component sizes are U-Net 867.6M, VAE 83.7M, text encoder 340.4M and calibration branch 23.4M. TR2M reports 18.960M trainable and 471.764M total prediction parameters. New RGB inputs additionally require Qwen3-VL-8B for captioning, approximately 9.3B parameters across both stages. Caching does not remove caption-generation cost; additional descriptions and fusion also require computation. SSAA's 768 auxiliary parameters are training-only. We claim parameter-efficient adaptation, not lower end-to-end cost than TR2M.

8、Hook in Eq. 3

Hook extracts the output of up_blocks[1], the second U-Net upsampling block. In the single-pass t=999 setting, F¹⁶ has 1,280 channels at H/16 × W/16 resolution. Hook introduces no trainable parameters.

9、Detaching both tensors

R and F¹⁶ enter separate calibration paths. Detaching both preserves their forward values and blocks gradients into the frozen backbone, while the calibration parameters remain trainable.

10、Rescale maps

These are pixel-wise scale A and shift B in R̂=A⊙R+B, applied in inverse depth. Each pixel has its own pair, unlike an image-wide scalar scale–shift pair.

Dear Reviewer,

We sincerely apologize for disturbing you. Last week we provided point-by-point responses to each of your ten comments, together with additional controlled experiments.

In particular, we added an incremental comparison that starts from a reconstruction of TR2M's training pipeline (released model, encoders and base losses, verified by reloading the official weights) and introduces our elements in two stages — the three depth-structured local descriptions with layered fusion, then SSAA — reporting δ₁ and AbsRel on NYUv2, KITTI, SUN RGB-D, Virtual KITTI 2 and DIML Outdoor at each stage. We also clarified the scope of Table 5 as a limited compatibility diagnostic, separated the Marigold GT-assisted reference from direct metric prediction, reported the complete inference footprint (1,314.975M parameters for depth prediction plus Qwen3-VL-8B for captioning, with caching acknowledged as moving rather than removing that cost), and added a component-wise ablation visualization in Appendix D.11, Fig. 8. The implementation questions on the Hook, the detach operation and the rescale maps are answered in the same response.

We would greatly appreciate your feedback on whether these responses adequately address your concerns, or whether any questions remain. As the discussion deadline is approaching, we would be grateful if you could let us know as early as possible, so that we have sufficient time to address any remaining point before the discussion period ends.

Thank you again for your valuable and constructive feedback, which has helped us clarify the methodological contributions and strengthen the experimental evidence.

Best regards,

The Authors

AuthorsOpenReview ↗

Follow-up on the additional controlled comparisons

Dear Reviewer,

As the discussion period is nearing its close, we would appreciate your feedback on our rebuttal and additional experiments.

To address your request for controlled comparisons, we evaluated incremental additions of local descriptions with layered fusion and SSAA to a TR2M adaptation. We also compared flat and layered fusion using the same frozen backbone and textual inputs. With SSAA disabled, the latter comparison reduces mean zero-shot AbsRel from 0.3207 to 0.3012, providing evidence for the fusion design under matched inputs. We additionally clarified the Marigold evaluation protocols and the parameter accounting for depth prediction and caption generation.

Could you please let us know whether these additions address your main concerns, or whether any substantive issue remains unresolved? We would be happy to clarify any remaining point. If the additional evidence satisfactorily addresses your concerns, we would be grateful if you could reconsider your assessment and score in light of these results.

Thank you again for your time and consideration.

Reviewer_huAaOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper adapts a frozen relative-depth model to estimate metric depth. It builds on a diffusion-derived depth backbone and uses VLM-generated descriptions to help calibrate depth. These descriptions cover both the whole scene and its foreground, midground, and background. An auxiliary module, SSAA, provides scene-scale supervision. The main contributions are language conditioning organized by depth, parameter-efficient calibration, and auxiliary supervision based on scale anchors.

The evaluation covers depth estimation and zero-shot transfer across multiple datasets. It also includes component ablations, comparisons of text organization strategies, and tests with missing or mismatched captions. The results support the effectiveness of the proposed components, but questions remain about what drives the gains and how much computation the method requires.

Critical strengths

Language-assisted calibration with a frozen backbone is a valuable adaptation strategy.

The method preserves the pretrained depth backbone and introduces language and scale supervision through relatively small trainable components, avoiding full-backbone fine-tuning. This provides a potentially useful route for adapting existing relative-depth models to metric prediction. Organizing language by depth layers also offers an explicit and testable hypothesis for connecting scene semantics with scale estimation. This is the paper’s strongest methodological aspect.

Critical weaknesses
  1. The methodological contribution beyond prior text-guided calibration is not clearly distinguished from the change in depth backbone

The paper needs to explain more clearly how its method differs from prior work such as TR2M. As described, both methods use image and text information to calibrate relative depth with spatially varying scale factors. The current experiments leave open a simpler interpretation: the method adapts an existing feed-forward, image-and-text-guided pixel-wise rescaling approach to a diffusion-derived backbone, then adds richer descriptions and auxiliary scale supervision. It is not yet clear whether the calibration mechanism itself is fundamentally different. The paper should identify which limitations of prior methods it addresses beyond changing the backbone and its features, and which gains depend on the proposed design.

In particular, foreground, midground, and background descriptions do not seem specific to diffusion-derived features. They could potentially be used in methods such as TR2M as well. This transferability is not a weakness, but the benefit of layered descriptions should be separated from that of the overall framework. Would the performance gap remain if prior methods used the same descriptions and comparable scale supervision? Would the proposed calibration module outperform a TR2M-style module on the same diffusion-derived backbone? These controlled comparisons are needed to distinguish the benefits of stronger depth features, richer text, and the calibration mechanism itself. Without them, the methodological novelty is difficult to assess. My main concern is identifying what drives the gains and what is methodologically new, rather than efficiency reporting alone.

  1. The proposed mechanisms and reliability under caption noise remain insufficiently validated

The ablations show that SSAA and depth-organized language improve the final results.But they do not fully explain why these gains occur. For SSAA, the evidence mainly concerns overall depth errors, rather than global scale bias in individual images. The scale anchors are also linked to the scale ranges of the training domains. It is unclear how much SSAA benefits from modeling scene scale and how much it relies on coarse domain priors.

For language organization, using the same branches and architecture does not ensure that the descriptions cover the same entities, contain the same relational information, or have similar lengths. Richer semantic content could therefore explain part of the improvement. Finally, valid formatting and qualitative examples do not show how accurate the foreground, midground, and background assignments are. The mismatch tests are useful, but the large performance drop under cross-domain mismatches also raises reliability concerns. More realistic, localized errors in depth-layer assignments remain insufficiently tested.

These gaps weaken the claim that the gains specifically come from reliable depth-layer descriptions and scene-scale modeling. Broader semantic information or dataset-domain priors may also play an important role.

Questions for authors
  1. Contribution clarification and essential controls

What changes does the paper introduce to the calibration mechanism relative to TR2M-style image-and-text-conditioned pixel-wise scale calibration, beyond changes to the depth backbone, descriptions, and auxiliary supervision? Could the layered descriptions also be used in a TR2M-style method? Do the authors have either of the following comparisons to isolate the contribution:

(a) a prior calibration method supplied with the same layered descriptions and comparable scale supervision; (b) the proposed and TR2M-style calibration modules evaluated with the same diffusion-derived backbone and textual inputs

or other results the authors feel are appropriate for this question.

  1. Is there evidence that SSAA directly reduces image-level global-scale bias, rather than only improving aggregate depth metrics? In particular, do its benefits persist across different scene scales within the same dataset domain?

  2. When entities, relations, and approximate text length are held constant, does correct depth-layer organization still outperform random grouping or layer swapping? Additionally, have the authors audited layer-assignment errors in generated captions or evaluated captions with correct overall scene context but incorrect local depth assignments?

I I am open to revising my rating if the authors adequately address my concerns.

4 comments

Thank you for asking which gains come from the calibration design. We added two complementary controls: a fusion comparison with identical backbone features and text, and an incremental transfer of our additions to a TR2M adaptation.

1. Contribution relative to TR2M

We share TR2M's inverse-depth scale–shift operator, R̂=A⊙R+B, and pseudo scale–shift loss, and credit this prior formulation. Our contribution comprises native diffusion-feature reuse, depth-structured local-language fusion, and SSAA scene-scale supervision. The fusion uses three weight-sharing cross-attention branches with layer embeddings and a zero-initialized residual projection. At initialization, only the local-language residual is zero; global text conditioning remains active. SSAA links learned feature anchors to fixed scale centres and adds no inference-time head. Table 6 measures component increments within our framework; the new controls below more directly examine their contribution beyond changing the backbone.

2. Matched comparison of fusion modules

To address the calibration mechanism directly, we compared a flat TR2M-style fusion control with our layered fusion, with SSAA disabled. Both use the same frozen diffusion backbone, F16 features, relative disparity, global description, denoising setting and training objective. Both receive the same three local descriptions encoded into the same token memory, with identical content and token counts; neither input is pooled to a single token or truncated.

Results (flat / layered):

  • KITTI: δ₁ 0.9630 / 0.9655; AbsRel 0.0593 / 0.0580
  • HyperSim: δ₁ 0.3136 / 0.3399; AbsRel 0.4082 / 0.4006
  • DIML Outdoor: δ₁ 0.3490 / 0.4306; AbsRel 0.5042 / 0.3720
  • Eight zero-shot datasets, mean: δ₁ 0.5383 / 0.5524; AbsRel 0.3207 / 0.3012

The eight-domain mean improves by 1.41 percentage points in δ₁ and 6.1% relative in AbsRel. Gains vary in size, with a particularly large reduction on DIML. This supports the tested fusion design under matched visual and textual inputs, beyond the effects of stronger backbone features or richer descriptions.

The flat control does not include TR2M's dual-level contrast. Thus, this experiment compares fusion modules under a common objective; the transfer experiment below retains the original TR2M losses. The fusion control evaluates the design as a whole, without isolating layer embeddings, branching or fixed slot identities individually.

3. Incremental additions to a TR2M-style method

We built a controlled adaptation using TR2M's released model and encoders and its original base losses. Loading the official checkpoint matched all parameters, verifying model compatibility. We then trained three configurations on the same data: B, the global-description baseline; L, B plus the three local descriptions and layered fusion; and S, L plus SSAA. The backbone, image/text encoders and original losses are retained throughout.

δ₁ (B / L / S):

  • NYUv2: 0.9280 / 0.9313 / 0.9357
  • KITTI: 0.9480 / 0.9512 / 0.9544
  • SUN RGB-D: 0.5982 / 0.6154 / 0.6359
  • Virtual KITTI 2: 0.8121 / 0.8383 / 0.8409
  • DIML Outdoor: 0.4008 / 0.4230 / 0.4551

AbsRel (B / L / S):

  • SUN RGB-D: 0.3785 / 0.3651 / 0.3561
  • Virtual KITTI 2: 0.1292 / 0.1130 / 0.1156
  • DIML Outdoor: 0.4110 / 0.3845 / 0.3555

Both steps improve δ₁ on all five evaluated datasets. From B to S, relative AbsRel decreases by 5.9%, 10.5% and 13.5% on the three zero-shot datasets, respectively. The SSAA step is not uniformly beneficial: Virtual KITTI 2 AbsRel increases by 0.0026 while δ₁ improves.

This controlled adaptation establishes incremental benefits within TR2M; it is not a reproduction claim or a full-system comparison against the enhanced TR2M. The first step measures descriptions and fusion jointly, while the second isolates added scale supervision. The improvements therefore show that these additions contribute beyond replacing the relative-depth backbone.

This response addresses the reviewer's questions about image-level scale calibration, depth-layer organization and caption reliability.

1. Direct measurement of image-level scale bias

We compare checkpoints trained with and without SSAA on the same 2,106 images: 1,306 from the in-domain NYUv2/KITTI splits and 100 from each of eight zero-shot datasets, using identical masks, crops and depth ranges.

For image i, let e_i(p)=log(predicted depth)−log(GT depth) and b_i be its mean over valid pixels. The mean squared log error decomposes into b_i² plus the mean squared residual after subtracting b_i. Thus, b_i measures the global log-scale offset separately from spatial variation. Adding SSAA reduces mean |b_i| from 0.140 to 0.132 and the mean absolute discrepancy between predicted and GT log-median depth from 0.146 to 0.137. These measurements directly support improved image-level scale calibration on the evaluated subset, beyond aggregate depth accuracy.

2. Scene scale and domain priors

The fixed log-depth centres [0.33, 1.03, 2.43] are derived from training data; the feature anchors are learned. Each image receives a soft target based on its GT median depth, rather than a dataset label.

To examine variation within domains, we divide each of the ten datasets into four equal-frequency bins using its own GT median depth. SSAA reduces absolute scale bias in 24 of the 40 bins, with larger pooled gains for farther scenes. This provides evidence of benefits within multiple domain-specific scale ranges, although they are not uniform. Training-derived centres can still encode domain-related priors. Without a control using domain-only supervision, these results do not fully separate scene-scale learning from such priors.

3. Matched-content fusion and layer organization

The new training control in our first response holds descriptions and token counts fixed: with SSAA off, layered fusion improves eight-domain mean AbsRel from 0.3207 to 0.3012 and δ₁ from 0.5383 to 0.5524 over flat fusion. This directly addresses the concern that richer textual content alone could explain the comparison.

We distinguish this result from inference-time slot permutation. Permuting the same descriptions changes AbsRel by +0.0019 on NYUv2, +0.0001 on KITTI and −0.0030 in the eight-domain mean. The training comparison supports the fusion design as a whole; permutation tests the sensitivity of an already trained model to slot identities while preserving the captions' depth-related wording. These findings can coexist: a useful layered architecture need not depend strongly on a fixed foreground/midground/background slot binding.

Accordingly, the evidence supports layered fusion under matched content, but does not establish that correct slot order itself causes the gain. A training comparison with correct versus random assignments within the same architecture would isolate that question. Appendix D.7 varies generated content and cannot supply this control.

4. Caption screening and assignment accuracy

Our automated textual-consistency screen flags 26/37,845 NYUv2 and 8/43,902 KITTI captions for explicit distance-language contradictions, such as “far away” in a foreground description. These are screen hits, not verified assignment-error rates: false positives and missed errors are both possible. Grounded-SAM localization provides qualitative checks but cannot itself determine whether an entity belongs to the correct depth layer. The current checks therefore do not quantify naturally occurring layer-assignment accuracy; an image-grounded audit remains an evidence gap.

5. Robustness to local description-slot swaps

We keep the global description fixed and swap a pair of complete local descriptions. Swap probabilities of 0.25, 0.5 and 1.0 specify the proportion of affected images, not the proportion of incorrect entities within an image. Adjacent swaps increase NYUv2 AbsRel by 0.0003, 0.0007 and 0.0016; the foreground–background swap increases it by 0.0040. KITTI changes remain below 0.0004, while eight-domain mean changes range from −0.0048 to −0.0003.

These results show limited aggregate sensitivity to the tested description-slot swaps. Because the descriptions retain their original statements, the test does not measure robustness to false object-depth relations or hallucinated entities. We therefore restrict the reliability claim to the tested perturbations; robustness to incorrect local statements under correct global context remains unestablished.

Thank you for the detailed rebuttal and additional experiments, which address most of my concerns. My remaining question concerns inference speed and memory usage. One practical motivation for converting relative depth to metric depth is to support applications requiring real-time metric depth estimation, where many large metric-depth models are too slow. TR2M claims to support real-time inference, so it would be helpful to report this method’s latency/FPS and peak GPU memory usage on comparable hardware, distinguishing depth prediction alone from the full pipeline including caption generation. If the method cannot operate in real time, could the authors clarify its primary intended applications and the practical benefits that justify the additional computational cost?

Thank you for this helpful follow-up. We now report measured latency, FPS and peak GPU memory, separating depth prediction with available descriptions from the complete RGB-to-depth pipeline.

Measurement protocol. We benchmarked both methods on the same NVIDIA RTX 4090 (48 GB), using batch size 1 and input images of 434×560. The matched TR2M comparison uses FP32. We measured 100 images after 10 warm-up requests with synchronized wall-clock timing. Timings include preprocessing, text encoding, required transfers and output processing, but exclude model initialization and disk I/O. Peak allocated GPU memory includes resident model weights and inference buffers.

Measured deployment cost.

TR2M, cached descriptions, FP32: 67.07 / 68.48 ms mean/P95, 14.91 FPS, 2.40 GiB peak memory.

Ours, cached descriptions, FP16: 129.57 / 138.24 ms mean/P95, 7.72 FPS, 3.04 GiB peak memory.

Ours, cached descriptions, FP32: 164.40 / 165.79 ms mean/P95, 6.08 FPS, 5.82 GiB peak memory.

Ours, complete RGB-to-depth pipeline (amortised, 4-GPU captioner): 2,420.90 / 2,763.90 ms mean/P95 amortised, 0.41 images/s, 16.5 GiB per GPU (19.38 GiB if the captioner and depth models are co-located).

Cached descriptions are text strings: text encoding, the relative-depth backbone and calibration are still executed. No image features or relative-depth predictions are cached. The complete pipeline additionally generates the global and local descriptions with Qwen3-VL-8B in bfloat16 without quantization, while the depth pipeline runs in FP16. The captioner is replicated across four GPUs and processes images in parallel, so the 2.42 s per-image figure in the table is an amortised cost under that four-GPU throughput rather than a single-image latency; one image takes about 9.0 s to caption and 0.17 s to convert to metric depth. Each model stays resident on its device with no offloading, and all per-request transfers are included in the reported timings.

Under the matched FP32 protocol, our depth path requires 2.45× the latency and 2.43× the peak memory of TR2M. The FP16 row is an additional profile of our implementation, reducing latency by 21.2% and peak memory by 47.8% relative to our own FP32 configuration. It is reported separately from the matched comparison. The TR2M measurement covers its cached-description path; we do not interpret it as a complete caption-generation pipeline.

Real-time suitability and intended applications. The present implementation does not meet a 30-FPS video-rate target, including when descriptions are cached. The complete pipeline has a single-image latency of about 9.2 s, equivalent to an amortised throughput of one image per 2.42 s on four GPUs. We therefore position it for offline metric-depth estimation and geometric analysis of collected imagery, particularly workflows that already use a frozen diffusion-depth backbone and can tolerate this preprocessing latency. Caching moves description generation to an earlier stage; it does not eliminate the initial cost for a new image. The profiled depth backbone already uses a single deterministic forward pass.

Benefits relative to the computational cost. The practical benefit is adding metric calibration while retaining pretrained backbone weights and avoiding full-backbone fine-tuning. Our system-level results improve over TR2M on five of the eight zero-shot datasets, with regressions on three, so the accuracy–cost trade-off remains domain-dependent.

The incremental cost of the fusion design is separately quantified by the matched-backbone, matched-text comparison with SSAA disabled. Under FP16 with the depth path only, batch size 1 at 434×560 and 100 images after 10 warm-ups, layered fusion reduces mean zero-shot AbsRel from 0.3207 to 0.3012 (6.1% relative) while adding 1.71 ms and 0.004 GiB over flat fusion. This supports the fusion design within a fixed backbone setting; it does not account for the full system's additional cost over TR2M. SSAA supplies training supervision without an inference-time head.

Our efficiency claim is therefore parameter-efficient adaptation of a large frozen backbone. Native-feature reuse avoids an additional calibration visual encoder, while the measured total latency and memory define the deployment requirements of the current implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.