acceptodds
Under review as a conference paper at ICLR 2027

Pumpire: Unified Benchmark for Metric Distance Estimation

Abstract

We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation models, with or without depth priors. In contrast to previous approaches that normally evaluate depth and camera intrinsics separately or evaluate point-clouds with geometric similarity metrics, which cannot directly reflect models' point-to-point distance estimation capability, Pumpire directly assess point‑to‑point distances from the reconstructed geometry. To this end, we collect a large-scale and diverse dataset (pumpire-6k) comprising 100 real-world scenes, each annotated with physically measured point-pair distances and containing 64 frames, for a total of 6,400 frames. Building on this dataset, we establish a holistic evaluation protocol that covers both image- and video-level 3D foundation models and enables direct assessment of point-pair distance errors and cross-setting comparison. We conduct extensive experiments across 29 baseline configurations of representative 3D foundation models and provide a comprehensive analysis of the results. By offering this benchmark, we target the more fundamental ability to perceive and estimate physical scale in the reconstructed 3D space, which prior evaluation protocols have largely overlooked.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D foundation models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oU9vOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Pumpire, a benchmark for evaluating the metric point-pair distance estimation of 3D foundation models. The authors capture 100 real-world scenes (40 indoor, 60 outdoor) with a RealSense D435, place two stickers in each scene, and measure the distance between their centers with a tape measure. Each sequence is subsampled to 64 frames, and sticker centers are annotated per frame with a CoTracker3-assisted, human-verified pipeline. The protocol back-projects the two annotated pixels using each model’s predicted depth and intrinsics (GT intrinsics for completion models) and compares the resulting distance with the measured one, using AE, RE, log error, threshold accuracy, and a cross-view consistency metric (RSD). Scenes are split into 76 “normal” and 24 “challenging” based on the cross-view consistency of raw sensor depth. The authors evaluate 29 configurations spanning image- and video-level estimation and completion models, and report several findings: depth and intrinsics benchmarks do not predict point-pair accuracy; raw sensor depth is a strong baseline on normal scenes but is overtaken by learned models on challenging ones; completion models are more accurate but fail catastrophically on challenging scenes; and denser depth priors do not always help.

Critical strengths
  1. The task is real, and the ground truth is independent of the sensors being evaluated. Metric distance between points is what downstream systems actually need, and tape-measured GT avoids the failure modes of sensor-derived GT on transparent, reflective, and thin structures. This is the core idea that makes the benchmark worth pursuing.
  2. The broad evaluation, including a raw-sensor baseline, gives a useful picture of the field. The results show that raw sensor depth is already hard to beat on normal scenes, while learned models and sensors behave very differently on challenging ones. This clearly indicates where the open problems lie, even though the paper does not fully draw this conclusion itself.
Critical weaknesses
  1. The positioning is not strong enough. The paper motivates the benchmark with a general claim about robotics and autonomous driving, but its own results show that normal scenes are largely solved: raw sensor depth beats all estimation models there, and existing laser-scanned datasets (e.g., ScanNet++, ETH3D) already provide accurate GT for such scenes. The real need for independently measured GT lies in conditions where sensors fail. In my view, the most important of these is outdoor severe weather. In rain, snow, ice, and flooding, the scene geometry itself changes: water accumulates on surfaces, ice forms on roads, water levels rise, snow and mud shift. LiDAR and active depth sensors are severely degraded; for example, wet surfaces such as roofs or vehicles can return few or no points. These conditions are not rare. For vehicles, drones, and construction robots in northern regions, snow and ice cover the environment for much of the year, and the system must perceive these conditions as they occur. Measurements taken before or after the event cannot serve as GT, because the geometry of interest exists only during it. This should be the motivation for the benchmark.
  2. The scale, especially of the challenging subset, is too small. Each scene has a single point pair, so the benchmark rests on 100 measurements, and the challenging subset has only 24 scenes. No confidence intervals are reported, so many rank differences are likely within noise. The challenging split is also defined by sensor-depth consistency, which partly predetermines findings about the sensor and the completion models that take sensor depth as input. The authors should use a small but sufficient set of normal scenes to confirm that current methods work there, and devote most of the effort to substantially more challenging scenes, particularly transparent and reflective surfaces and severe weather.

Response to the author's rebuttal:

  • Their plan is clear, I think with the expansion, it could be a stronger work in future
Questions for authors

Expansion plan (this would most affect my score):

  1. Can you outline a concrete plan to expand the challenging subset, especially transparent and reflective surfaces and outdoor severe weather (rain, snow, ice, flooding) with in-situ measurement? A small pilot set would strengthen the case considerably.
  2. Will the dataset and evaluation code be released, and under what license?
4 comments

W1: Positioning of the benchmark and expansion to sensor-failure scenes such as severe outdoor weather.

We thank the reviewer for this thoughtful suggestion.

(1) On normal scenes being largely solved. By proposing a benchmark that directly evaluates point-pair distance and unifies the evaluation of image- and video-level estimation and completion models, our goal is not to find the best ranging method, but to drive the progress of 3D foundation models. Pumpire aims to guide them toward reliable performance in real-world scenes, including normal ones, and ultimately beyond active sensors.

(2) On severe outdoor weather. We share the reviewer's view that severe outdoor weather, such as rain, snow, ice, and flooding, is critical for real-world perception and an important direction for future research. Pumpire takes the first step by focusing on common real-world scenes. To our knowledge, it pioneers point-pair distance evaluation across method families, including the raw sensor.

(3) On expanding to sensor-failure scenes. Collecting such scenes with in-situ measurement is a key next step, with a concrete plan given in Q1. Nevertheless, the current benchmark is already meaningful on its own.

W2: Limited scale, bias of the sensor-based difficulty split, and expansion of challenging scenes.

(1) Scale. We acknowledge that the scale is limited, but it is comparable to widely used benchmarks: Bonn, ETH3D, and DIODE in the MoGe-2 evaluation contain 26, 13, and 20 scenes, respectively, so even our challenging subset (24 scenes) is of similar size, and the full benchmark (100 scenes) is larger. Enlarging the challenging subset is indeed important, and we will pursue it in future versions.

(2) Difficulty split. We recognize that the sensor-based split may bias findings about the sensor and completion models. We therefore regroup all 100 scenes by surface type, classified by Claude Opus 5.5 from RGB images alone, and report RE:

ModelTransparent / reflective (18)Slender / hollow (13)Edges / steps / rocks (9)Repetitive-texture finishes (12)Ground (11)Flat matte (37)All (mean / median)
Raw sensor8.0524.2430.1020.0310.0310.0692.043 / 0.040
Metric3D v22.4290.1080.1730.1290.0880.0910.526 / 0.093
Depth Pro0.2580.5440.2260.2740.2470.1440.251 / 0.151
Depth Anything 30.2350.5210.2630.1190.1250.0670.187 / 0.092
Lingbot-Depth1.8102.3160.0880.0240.0210.0360.653 / 0.027
CAPA-VGGT4.1803.0480.0570.0160.0190.0371.172 / 0.022

Under this sensor-independent grouping, Findings 2 and 3 still hold. We will add this analysis to future versions of the paper.

(3) Expanding challenging scenes. We fully agree on its importance and will expand transparent, reflective, and severe-weather scenes in future versions (plan in Q1).

Q1: Expansion plan and pilot set (anonymous link).

We will substantially expand the challenging subset with the following scene categories:

  • Transparent and semi-transparent surfaces (anytime, indoors and outdoors): glass doors and windows, glass curtain walls, shop windows and storefronts, glass partitions and railings, glass staircases and skywalks, bus shelters, greenhouses, display cases and aquariums, glass tables and shelves, glass bottles, cups, and containers, frosted and patterned glass, acrylic and plastic sheets, plastic curtains and films, and translucent containers.
  • Reflective surfaces (anytime, indoors and outdoors): mirrors and mirrored walls, polished metal (e.g., elevator doors, stainless-steel kitchenware, equipment, and sculptures), glossy floors and tiles, lacquered furniture, glossy screens and appliances, car bodies and windshields, and mirrored building facades.
  • Rain and wet surfaces (on rainy days, especially during the rainy season): wet roads, sidewalks, and crosswalks, wet roofs and vehicles, wet stairs and ramps, rain-covered glass and windshields, wet construction sites, and scenes during active rainfall.
  • Standing water and flooding (during and after heavy rainfall): puddles on roads and sidewalks, waterlogged roads, parking lots, and courtyards, flooded low-lying areas and lawns, and water-covered steps.
  • Snow (in winter, or in high-latitude and high-altitude regions): snow-covered roads, sidewalks, and stairs, snow-covered vehicles, roofs, and vegetation, snow piles and drifts, partially cleared roads, and scenes during active snowfall.
  • Ice (in winter, or in high-latitude and high-altitude regions): icy roads and sidewalks, frozen ponds and lakes, icicles, frost-covered glass and vehicles, and ice-covered railings, stairs, and structures.
  • Fog and haze (on foggy mornings and in seasons with frequent fog): roads, parks, and building surroundings under fog or haze, which degrade both visual and active sensing.

We will capture these scenes following our original pipeline, enriched with the within-frame point pairs of varying difficulty and cross-frame point pairs suggested by Reviewer ycfH, with all distances measured in situ, to scale up the benchmark. As a pilot set, we have already collected 6 scenes: 3 with transparent surfaces, 2 with reflective surfaces, and 1 with a puddle, available at this anonymous link. The remaining categories depend on weather conditions and will be captured as conditions permit.

Q2: Release and license of the dataset and code.

Yes. We will publicly release all data, including the RGB-D sequences, camera intrinsics, tape-measured distances, and per-frame annotations, together with the full evaluation code, under the Apache License 2.0.

Reviewer_oU9vOpenReview ↗

I am Satisfied

Keep on the good work

AuthorsOpenReview ↗

Response to reviewer oU9v:

Thank you for your constructive suggestions. We will expand Pumpire as planned.

Reviewer_sxruOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a new metric depth/distance estimation evaluation benchmark consisting of 100 real-world scenes and 6.4k frames. It allows a unified evaluation of mono-depth, video-depth and depth completion. And it contains subsets with different evaluation hardness to assess when and where existing methods perform good or bad.

Critical strengths

The main advantage of the benchmark is to provide a unified data pool for evaluating different types of 3D foundation models.

The paper also conducts in-depth analysis on large amount of depth models.

The paper writing is clear and the experiments are organized.

Critical weaknesses

At this stage, it is hard for me to be convinced that the proposed benchmark by itself has significant advantage over the standard protocals of combining multiple existing benchmarks. The main reason lies in the lack of in-depth and organized comparisons to the standard benchmarks used in existing papers, and clear result of why some functions in the proposed benchmark cannot be replaced by others.

Though the 100 scenes are included with both indoor and outdoor scenes, there is no statistics to how diverse the scenes are and whether they are more diverse comparing to existing standard benchmarks. Without having concrete statistics on things like where the data are captured (geolocation etc.), why the proposed benchmark is more diverse than existing ones, it is hard to tell at what scale can we trust the results of the analysis.

We also dont know how redundant the 6.4K images in from the 100 scenes are, like there are 64 images per scene, without qualitative or quantitative analysis, it is hard to tell how diverse the benchmark is in terms of per-frame information.

Also the 6 findings presented are not very surprising to me from the big picture perspective, and as mentioned above, without clear comparison, it is hard to judge how much of them cannot be derived from existing benchmarks and how much of them can generalize to other datasets.

Questions for authors

This is more out of interests, there have been recently a line of research using VLMs for 3D understanding (Seed model series, Gemini, DepthVLM, DepthLM, VLM3, Youtu-VL), some of them shows different properties than traditional 3D foundation models, like much fewer flying points shown in DepthLM. It would be nice to include some discussions and investigations to VLM-based methods as well.

4 comments

W1: Advantage over combining existing benchmarks and organized comparison with them.

Pumpire offers two capabilities corresponding to the two motivations of our paper (Sec. 1):

(1) Why combining existing benchmarks cannot replace Pumpire. Pumpire directly evaluates point-pair distance, which combined depth and intrinsics benchmarks fail to predict (Finding 1).

(2) Organized comparison with existing benchmarks. Pumpire unifies all four settings, whereas existing benchmarks are designed fragmentedly across them:

BenchmarkImage-level estimationImage-level completionVideo-level estimationVideo-level completion
KITTI✔✔✘✘
NYU v2✔✘✘✘
VOID✘✔✘✘
ETH3D✘✘✔✘
Tanks and Temples✘✘✔✘
ScanNet✘✘✘✔
7-Scenes✘✘✘✔
Pumpire (Ours)✔✔✔✔

W2: Diversity statistics and comparison with existing benchmarks.

We classify each scene with the Places365 scene classifier [1] (365 categories; 8 frames per video scene). To compare datasets of different sizes, we randomly sample k scenes from each dataset 1,000 times and report the average number of distinct categories and the category entropy (bits):

Dataset#Scenes#Categories ↑ (k = 13)Entropy ↑ (k = 13)#Categories ↑ (k = 20)Entropy ↑ (k = 20)
Bonn263.31.263.71.33
Spring378.42.7911.43.12
ETH3D139.02.97––
DIODE209.43.1112.03.45
NYUv2654*9.83.1613.33.53
Pumpire10010.23.2214.23.63

*Test images. ETH3D has only 13 scenes, so k = 20 is not applicable.

Pumpire shows scene diversity comparable to or higher than widely used depth benchmarks.
[1] Zhou, Bolei, et al. "Places: A 10 million image database for scene recognition."

W3: Redundancy among the 64 frames per scene.

We measure how much adjacent frames differ using RAFT optical flow [2], compared with two widely used video benchmarks, Sintel and Spring. We report the median flow magnitude normalized by the image diagonal (higher means larger change) and the co-visibility, i.e., the fraction of pixels visible in both frames (lower means less shared content):

#Seq / #FramesFlow / diagonal ↑Co-visibility ↓
Sintel23 / 1,0640.0110.899
Spring37 / 1,0000.0110.922
Pumpire100 / 6,4000.0360.851

Adjacent Pumpire frames move about 3× more and share less content than those of both benchmarks (vs. Sintel, Mann-Whitney U test: p < 1e-8 and p = 0.015), indicating that our frames are less redundant than the consecutive frames of standard video benchmarks. We will include this analysis in the supplementary material.
[2] Zachary Teed, and Jia Deng. "RAFT: Recurrent all-pairs field transforms for optical flow."

W4: Novelty of the findings, their derivability from existing benchmarks, and generalization.

To our knowledge, none of the six findings has been reported before, and even those consistent with intuition had not been quantitatively verified; Pumpire provides the first such evidence against physically measured ground truth. The findings fall into two groups:

(1) Findings that cannot be derived from existing benchmarks, as they rely on our protocol and data collection:

  • Direct point-pair evaluation reveals the depth–intrinsics coupling that separate depth and intrinsics benchmarks cannot capture. (Finding 1)
  • Physically measured ground truth remains reliable where sensor-derived ground truth fails. This enables a quantified comparison between raw sensor depth and learned models (Finding 2) and revealing the distinct robustness patterns of estimation and completion models (Finding 3).
  • Unified evaluation across image- and video-level estimation and completion, absent from fragmented existing benchmarks, reveals how multi-view context and depth priors affect distance accuracy and cross-view consistency. (Finding 6)

(2) Findings that may be obtained from other benchmarks, though, to our knowledge, Pumpire is the first to quantitatively verify them:

  • Denser depth priors do not necessarily help, as completion requires robustness to both sparsity and sensor noise. (Finding 4)
  • The benefit of additional depth priors for video-level completion depends on a model's ability to integrate metric cues across views. (Finding 5)

Q1: Discussion of VLM-based methods.

We thank the reviewer for this suggestion. We have evaluated four of the listed VLM-based methods (DepthLM, DepthVLM, Seed, and Gemini), together with Claude Fable 5.2; please see our response to Reviewer ycfH (W4) for the results and discussion. Youtu-VL is not evaluated, as its released code lacks the depth decoding procedure.

W1.

  1. For point pair distance, I agree that previous works didnt use it as the major metric. However, I dont agree that this is a unique metric only the proposed benchmark can provide. For image depth, it can be directly obtained from the existing benchmarks, while for video depth, it can be obtained from videos with intrinscs, extrinsics and depth.

  2. This table exactly reflects that combining existing benchmarks can provide every axes the proposed benchmark evaluates.

W2. This table shows that the proposed benchmark is not significantly more diverse than existing benchmarks, especially when we can combine existing benchmarks to increase diversity,

W3. This table shows that the proposed benchmarj has high covisibility (>0.8), meaning the redundancy is high for image depth.

W4. I totally agree that metric wise the submission investigated much more since the focus is on evaluation, however, these findings as I mentioned before, does not provide the level of insight that significantly changes the existing benchmarks and metrics. If these findings are the major contributions, I would encourage the authors to examine them also on existing benchmarks, if executable.

Overall, I really appreciate the efforts of the authors. However, I still would like to maintain my current scores based on the above reasons.

AuthorsOpenReview ↗

Response to Reviewer sxru

We sincerely thank the reviewer for the detailed follow-up.

W1: Uniqueness of point-pair distance evaluation and of the unified four-setting evaluation.

(1) Point-pair distance.

  • Granularity. To our knowledge, Pumpire is the first benchmark to evaluate point-pair distance at the pixel level; existing spatial reasoning benchmarks only assess object-level distances (Sec. 1).
  • Metrics. All seven metrics in our protocol (AE, RE, LE, δ1.05, δ1.10, δ1.25, and RSD) are proposed novelly by us for point-pair distance. To our knowledge, no prior benchmark has used them to evaluate point-pair distance.
  • Ground truth. Point-pair distances can be computed from a few existing datasets, but not reliably. As discussed in Sec. 3, real-world datasets derive ground truth from depth sensors, which are noisy, especially on transparent and reflective surfaces, while synthetic datasets suffer from a sim-to-real domain gap. Pumpire avoids both by measuring distances manually with a tape measure in real-world scenes.

(2) Combining existing benchmarks.

  • Unified and fair comparison. As stated in our first contribution, Pumpire compares the distance estimation of different model families fairly and systematically, on the same scenes, ground truth, and metrics. Existing benchmarks each cover only part of these families, so their results cannot be compared across all families.
  • Unification as a contribution. Unifying separate tasks within one framework is itself a recognized contribution. For example, PriorDA [1] unifies depth completion, super-resolution, and inpainting, which were previously handled by separate models (see its Table 1); VGGT [2] unifies the prediction of camera parameters, depth maps, point maps, and tracks in a single model.

[1] Wang, Zehan, et al. "Depth anything with any prior."
[2] Wang, Jianyuan, et al. "VGGT: Visual geometry grounded transformer."

W2: Diversity compared with combined existing benchmarks.

We share the reviewer's view that diversity is important, and we will substantially increase it in future versions, especially in challenging scenes (see the concrete plan in our response to Reviewer oU9v, Q1).

Our focus is to be, to our knowledge, the first benchmark that directly evaluates point-pair distance and unifies the evaluation of image- and video-level estimation and completion models. In terms of diversity, our goal is to be on par with or more diverse than individual existing datasets, rather than all of them combined, which is an inherently unfair comparison for a single benchmark. Pumpire achieves this goal (see W2 of our previous response).

W3: Redundancy among frames for image-level evaluation.

We acknowledge that frames within a scene share content, which is by design: multiple views are needed for cross-view consistency (RSD) and video-level models. This does not inflate image-level results, since metrics are averaged per scene and then equally across scenes (Eq. 6). Pumpire contains 100 scenes, more than Bonn, ETH3D, and DIODE (26, 13, and 20) used to evaluate MoGe-2, a representative image-level geometry model, and are thus sufficient for image-level evaluation.

W4: For "change the existing benchmarks and metrics", and contributions beyond the findings.

We sincerely thank the reviewer for recognizing the extent of our evaluation.

(1) For "change the existing benchmarks and metrics." First of all, Pumpire is not intended to replace existing depth and intrinsics benchmarks (metrics), but targets a different task, metric point-pair distance estimation. For this task, Finding 1 shows that existing benchmarks are insufficient (Tables 3, 7, 9), so point-pair distance needs to be evaluated directly.

(2) Contributions beyond the findings. The findings are not our only contribution. As listed in Sec. 1, our major contributions also include:

  • The first unified benchmark for directly evaluating metric point-pair distance across image- and video-level estimation and completion;
  • A real-world dataset with physically measured point-pair distances;
  • A unified evaluation of 29 baseline configurations that quantifies their distance estimation capability. Regarding the suggestion, examining the findings on other datasets cannot be completed within the discussion period, and we will include it in future versions.
AuthorsOpenReview ↗

Response to Reviewer sxru

Dear Reviewer sxru,

Thank you again for your time and the follow-up discussion on our paper. We have posted a further response to each of your remaining concerns.

We understand that this is a busy period. When convenient, we would be grateful if you could take a look and let us know whether our response addresses your concerns. If any questions remain, we would be glad to discuss them further before the discussion period closes.

Thank you for your consideration.

Best regards,

The Authors

Reviewer_ycfHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Pumpire, a unified benchmark for evaluating metric point-pair distance estimation by 3D foundation models. The authors collect and curate pumpire-6k, comprising 6,400 frames from 100 real-world scenes (40 indoor and 60 outdoor), with 64 frames per scene. The benchmark evaluates 29 baseline configurations covering both image-level and video-level geometry estimation and completion. Its protocol back-projects predicted depth using the associated camera intrinsics and compares the resulting 3D point-pair distances with physical measurements, assessing both distance accuracy and cross-view consistency. The experiments show that accurate depth estimation does not translate into accurate point-pair distance estimation. This raises new challenges for future 3D foundation models.

Critical strengths
  1. The research problem is important. Assessing physical scale in reconstructed 3D space is particularly important to robotics and other real-world applications. Evaluating point-pair distances is a practical way to assess this capability.

  2. The insights are clearly presented. The authors highlight the key findings throughout the paper.

  3. The authors physically measured ground-truth point-pair distances. Compared to distances derived from sensor depth, this enables evaluation across diverse surface properties and viewing conditions.

Critical weaknesses
  1. While this paper presents a new task and benchmark, some of the findings seem intuitive and expected. Poor performance by existing 3D models in challenging cases is generally expected, as their depth predictions are often unreliable on transparent or semi-transparent surfaces, such as windows.

  2. The comprehensiveness of the benchmark is limited. Although the dataset covers 40 indoor and 60 outdoor scenes, each consisting of 64 frames, the evaluation uses only one annotated point pair per frame, with the same physical pair tracked across all frames in each scene. This limits the number of distinct point pairs evaluated. It would be interesting to include point pairs of varying difficulty within each frame, as well as cross-frame point pairs (e.g., one point in frame 1 and another in frame 2). Increasing the benchmark diversity could reveal further insights into the capabilities of 3D foundation models.

  3. Although the paper provides some preliminary evidence of depth–intrinsics coupling, more analysis is needed to clarify when 3D models produce reliable depth estimates but inaccurate point-pair distances, and when distance estimation fails because the depth estimates themselves are unreliable.

  4. The evaluation does not include vision-language models (VLMs) [1, 2], which can also predict 3D attributes. Extending the evaluation to include VLMs can provide more insights. For video-level geometry estimation, including models like VGGT [3] and Pi3 [4] would also improve the evaluation completeneee.

[1] Yeh, Ching-Feng, et al. "Depthlm: Metric depth from vision language models."

[2] Cai, Zhipeng, et al. "VLM3: Vision Language Models Are Native 3D Learners."

[3] Wang, Jianyuan, et al. "Vggt: Visual geometry grounded transformer."

[4] Wang, Yifan, et al. "π3\pi^ 3: Permutation-Equivariant Visual Geometry Learning."

Questions for authors

Please see weaknesses.

4 comments

W1: Some findings seem intuitive and expected. (Poor performance on challenging cases)

Indeed, poor performance on transparent surfaces is expected for both learned models and sensors. Our findings instead concern how different model families behave on these scenes, which, to our knowledge, has not been examined before: sensor depth versus learned 3D foundation models (Finding 2) and distinct robustness patterns of estimation and completion (Finding 3).

W2: Limited number of distinct point pairs; within-frame and cross-frame pairs.

We thank the reviewer for this insightful point and would appreciate the suggestion of within-frame pairs of varying difficulty and cross-frame pairs, which would make the benchmark more informative. Since each pair requires on-site tape measurement, such labor-intensive collection cannot be completed within the dicussion period; we will include both types in future versions of the benchmark.

W3: Insufficient analysis of when distance fails due to depth versus intrinsics.

We evaluate the six image-level estimation models that predict intrinsics on the 76 normal scenes, splitting frames by whether depth is reliable (both points within 10% of sensor depth) and whether intrinsics are reliable (hFoV error ≤ 5°, as in Table 11), and report the distance failure rate (RE > 10%):

Depth reliableDepth unreliable
Intrinsics reliable15.5%83.3%
Intrinsics unreliable49.4%62.1%
  • Reliable depth, inaccurate distance: occurs when intrinsics are unreliable and thus mismatched with the correct depth (failure rate 15.5% → 49.4%).
  • Failure due to unreliable depth: distance fails even with reliable intrinsics (83.3%).
  • Coupling: with unreliable depth, unreliable intrinsics lower the failure rate (83.3% → 62.1%), i.e., the two errors partially compensate, consistent with Table 9.

W4: Evaluation of VLMs, VGGT, and Pi3.

We include the evaluation of five VLMs on all 100 scenes:

  • VLM-based depth models: DepthLM and DepthVLM, whose released weights cannot directly predict point-pair distance. They predict depth at the two pixels and a dense depth map, respectively, which are back-projected with ground-truth intrinsics since they do not predict intrinsics.
  • Frontier VLMs: Seed 2.1 Pro, Gemini 4 Argon, and Claude Fable 5.2, which directly output the point-pair distance.

The remaining models are not included because VLM3 has not released its weights, and VGGT and Pi3 predict geometry only up to an unknown scale; the metric-scale successor of Pi3, Pi3X, is already evaluated in Tables 4 and 6.

For reference, we report MoGe-2 back-projected with its predicted intrinsics (our standard protocol, Table 3) and with ground-truth intrinsics:

ModelProtocolAE↓RE↓LE↓δ1.05↑δ1.10↑δ1.25↑
MoGe-2Depth + Pred intrinsics0.1830.2690.21721.6138.3661.62
MoGe-2Depth + GT intrinsics0.1740.2490.21416.9531.5862.06
DepthLMDepth + GT intrinsics0.5541.0370.43310.0020.5946.56
DepthVLMDepth + GT intrinsics0.3820.6360.39411.4421.5243.53
Seed 2.1 ProDirect distance1.0291.4770.5365.6413.1427.38
Gemini 4 ArgonDirect distance0.1850.3430.22019.8436.3369.38
Claude Fable 5.2Direct distance0.2830.5580.31314.529.855.2

All results are averaged over all 100 scenes. Gemini achieves accuracy comparable to dedicated 3D foundation models (e.g., similar AE and LE to MoGe-2 and a higher δ1.25), whereas the other VLMs lag behind.

Thank you to the authors for the response, which partially addressed my concerns, especially W3 and W4. I encourage the authors to include these results in their revision. For W1 and W2, I believe incorporating more within-frame and cross-frame pairs will enhance the comprehensiveness of the benchmark and potentially yield more insights. It's understandable that such data is hard to collect, and this is exactly where I think the value of the proposed benchmark lies. Therefore, I will keep my score.

AuthorsOpenReview ↗

Response to Reviewer ycfH

We sincerely appreciate your kind support and understanding. In our final revision, we will further improve the paper by incorporating more within-frame and cross-frame pairs.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Doubts about the comments from Reviewer sxru

Dear ACs and PCs,

Thank you for your efforts in ensuring the quality of the reviews for our paper. We would like to briefly bring to your attention the remaining concerns of Reviewer sxru, who maintains the score. We respectfully believe that the assessment involves a misunderstanding and an unfair standard:

  • Misunderstanding of our goal (W4). The reviewer expects our findings to change existing benchmarks and metrics. However, Pumpire does not aim to replace them; it targets the task of metric point-pair distance estimation, for which Finding 1 shows that existing benchmarks are insufficient. Beyond the findings, our contributions also include the benchmark, the dataset, and the unified evaluation of 29 configurations.
  • Unfair comparison (W2). The reviewer compares the diversity of Pumpire, a single benchmark, against all existing benchmarks combined, a standard no single benchmark can meet. Compared with individual datasets, Pumpire is on par or more diverse.

We have also clarified the remaining concerns on obtaining point-pair distance from existing benchmarks (W1) and on frame redundancy (W3) in our response.

Therefore, we believe that our contributions to metric point-pair distance estimation are sufficient, as acknowledged by Reviewer ycfH and oU9v.

Best regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.