- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents DepthEvidence, a 4B multimodal language model that uses its own metric-depth predictions as object-aligned evidence for language generation. The method contains a camera-conditioned dense decoder for full-resolution metric-depth prediction and a dense-to-language interface that pools predicted depths and decoder features within referenced object regions. The resulting descriptors are converted into continuous geometry tokens anchored to explicit object identifiers. Auxiliary objectives supervise the recoverability of object-level metric statistics before and after language-context interaction, while a ranking loss encourages correct relative ordering. The paper additionally introduces Depth-VQA, which organizes depth understanding into dense metric prediction, instance-level metric measurement, and compositional reasoning involving relative relations, numerical gaps, ranges, thresholds, and spatial filters. Across nine dense-depth datasets, DepthEvidence obtains the highest average among the evaluated methods while remaining competitive with specialized estimators. It also achieves the strongest reported aggregate results on instance-level estimation and the two Depth-VQA reasoning tracks. Matched no-interface comparisons, supervision ablations, agent baselines, and general-VQA evaluations are used to analyze the contribution and capability retention.
Critical strengths
1.The paper identifies and addresses an important missing interface between geometric prediction and language reasoning. Existing depth-aware VLMs may predict depth and text in parallel without ensuring that language generation actually uses the predicted metric field. DepthEvidence addresses this gap through object-aligned geometry tokens tied to explicit identifiers and supervised for metric recoverability after contextualization. This is a focused and technically meaningful contribution rather than a superficial combination of two tasks.
2.The empirical study includes strong matched controls and covers prediction, reasoning, and capability retention. The comparison between the full model and “DepthEvidence w/o interface” isolates the interface more cleanly than comparisons against unrelated baselines. The full interface improves Relative Overall accuracy from 0.5838 to 0.6270 and Metric Overall accuracy from 0.7692 to 0.8504. Table 6 further decomposes the effects of raw-depth, contextual-depth, and ranking supervision. The paper also evaluates nine-dataset dense-depth generalization, agent pipelines, general VQA retention, and spatial understanding, providing evidence that the gains are not obtained by simply sacrificing the base model’s broader capabilities.
Critical weaknesses
1.The metric reliability and independence of Depth-VQA require stronger validation. For outdoor data, the paper states that sparse depth is first completed so that every image has a dense depth map. Object means, percentiles, ranges, and threshold decisions are then derived from these completed maps. The main paper does not identify the completion method, quantify its metric error, or explain how uncertainty in unobserved regions is propagated into benchmark labels. Manual verification can identify incorrect masks or inconsistent questions, but it may not establish that a generated object mean such as 1.31 m is metrically correct. The benchmark is also relatively small, with 757 Level-1 instances, 370 Relative questions, and 376 Metric questions. The paper should clarify strict separation between training and benchmark scenes, instances, and generated question patterns, and ideally report uncertainty or confidence intervals. This matters because the paper’s main reasoning claims depend on the reliability and independence of these labels.
2.The causal role of the model’s own predicted dense depth is not fully isolated. The interface descriptor combines decoder features, box coordinates, and summarized predicted-depth statistics. Table 6 shows that adding the interface with answer supervision alone produces almost no improvement; most of the measured gain appears after direct supervision of the geometry tokens with ground-truth object-depth statistics. This is consistent with the proposed method, but it does not by itself establish that inference-time answers depend on the model’s predicted depth map. A stronger causal test would separately evaluate depth-statistics-only and decoder-feature-only interfaces, replace the predicted depth with shuffled or perturbed depth, and compare predicted-depth evidence with a ground-truth-depth upper bound. Without such interventions, it remains possible that the interface relies substantially on decoder features, object/location priors, or task-specific instruction tuning rather than the explicit metric field emphasized in the paper’s central claim.
Questions for authors
1.What method is used to complete sparse outdoor depth when constructing Depth-VQA, and how is the metric accuracy of the completed values validated? In particular, how are object-level means and 5th/95th percentiles handled in regions with little or no original sensor support? Please also confirm whether benchmark scenes, object instances, and generated questions are strictly disjoint from the instruction-tuning data.
2.Can the authors provide a causal intervention demonstrating that the answers depend on the predicted dense metric field? For example, what happens if the predicted depth statistics are shuffled across objects, perturbed in scale, or removed while retaining decoder features? Conversely, what is the performance of a depth-statistics-only interface and of an interface supplied with ground-truth depth? These results would materially affect my confidence in the paper’s central claim.
3.How many independent linguistic templates or generation patterns are represented in the benchmark, and how sensitive are the results to paraphrasing or altered numerical ranges? Since training and evaluation questions are synthetically generated, evidence of robustness to unseen wording and constraint compositions would help distinguish geometric reasoning from template adaptation.
3 comments
Dear Reviewer WEPD,
Thank you again for your constructive feedback. In our rebuttal, we added a fixed-checkpoint depth-statistics shuffling experiment, statistics-only and features-only ablations, and a paraphrase evaluation. We also clarified the depth-completion procedure and scene/sequence separation.
We would greatly appreciate your thoughts on the new evidence, particularly regarding the inference-time use of predicted depth. We are happy to discuss any remaining concerns.
Thank you for the additional experiments and clarifications.However, the rebuttal describes held-out sensor validation, but I could not find the resulting errors or coverage rates in the supplementary material. What are these results, particularly for object regions with little sensor support?
Thank you for your constructive feedback on evidence use and benchmark validation.
W1. Label reliability and independence
Q1 details completion and validation, and 9WCi Q1 explains scene/video separation. Validation combines held-out sensor testing with manual inspection of metric geometry.
W2. Role of predicted depth
Fixed-Full shuffling covers all 746 Level-2 questions and reduces the Overall scores by 8.65 / 15.38 pp, adding evidence of the inference-time role of object-aligned statistics (Q2 / R5).
Q1. Completion and label validation
We express Waymo, Argoverse 2, nuScenes and DDAD depths as metric optical-axis Z. Within 0–250 m, we compare eight candidates (C1) in 20 m bins (final bin 240–250 m), using 400 frames per source–range. A base model is selected by equally weighted bin-level AbsRel; bin-specific replacements require adequate coverage and scene-resampling-supported RMSE improvement. Object statistics use valid depths within instance masks.
Each source has 100 independent-scene test frames, with 80% input / 20% held-out sensor points, plus contiguous-region withholding. After fixing selection, we compare MAE, RMSE, AbsRel, P95, δ₁ and coverage. We additionally inspect 120 frames per source (60 random, 60 targeted) using RGB, depth and LiDAR-overlaid point clouds for scale, boundaries and structure. Benchmark scenes and video sequences are excluded from both dense-depth and instruction-tuning training, following Appendix C.2 and the separation described in 9WCi Q1.
Table C1. Depth-completion candidates evaluated
Q2. Ablations and depth intervention
R1 reports separately retrained Stats-only / Features-only. We further fix Full and shuffle the six fields mean, standard deviation, minimum, maximum, center and log-mean as a bundle across objects before the projector. Features, boxes, location attributes, object IDs, images and questions retain their original inputs. All 746 questions are scored against the original labels.
R5 gives 54.05% / 69.66% Relative / Metric, drops of 8.65 / 15.38 pp from published Full, and gap δ₁ of 52.08% / 36.17%. This supports sensitivity to object-aligned statistics. Clean scores are the published references.
Table R5. Measured shuffle results against the Full reference
Note: Scores are percentages; drops are percentage points. Full clean values are the published references. All 746 questions received the intervention, with disjoint scoring groups of 370 / 234 / 48 / 94 questions. The table reports aggregate scores.
Q3. Wording and constraint generalization
We evaluate Full on original and rephrased versions of the same questions, preserving images, objects, numbers, semantics and answer format. This tests linguistic variation within the original numerical conditions and reasoning constraints. For example, “Which vehicle on the left is closest?” becomes “Among vehicles on the left side, select the one nearest to the camera.” Relative / Metric is 62.43% / 84.62%, drops of only 0.27 / 0.42 pp (R9), supporting stability under the tested rephrasings.
Table R9. Performance under meaning-preserving question rephrasing
Note: Images, objects, numbers, semantics and answer format are preserved. Changes use the supplied Full reference.