- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Pumpire, a benchmark for evaluating the metric point-pair distance estimation of 3D foundation models. The authors capture 100 real-world scenes (40 indoor, 60 outdoor) with a RealSense D435, place two stickers in each scene, and measure the distance between their centers with a tape measure. Each sequence is subsampled to 64 frames, and sticker centers are annotated per frame with a CoTracker3-assisted, human-verified pipeline. The protocol back-projects the two annotated pixels using each model’s predicted depth and intrinsics (GT intrinsics for completion models) and compares the resulting distance with the measured one, using AE, RE, log error, threshold accuracy, and a cross-view consistency metric (RSD). Scenes are split into 76 “normal” and 24 “challenging” based on the cross-view consistency of raw sensor depth. The authors evaluate 29 configurations spanning image- and video-level estimation and completion models, and report several findings: depth and intrinsics benchmarks do not predict point-pair accuracy; raw sensor depth is a strong baseline on normal scenes but is overtaken by learned models on challenging ones; completion models are more accurate but fail catastrophically on challenging scenes; and denser depth priors do not always help.
Critical strengths
- The task is real, and the ground truth is independent of the sensors being evaluated. Metric distance between points is what downstream systems actually need, and tape-measured GT avoids the failure modes of sensor-derived GT on transparent, reflective, and thin structures. This is the core idea that makes the benchmark worth pursuing.
- The broad evaluation, including a raw-sensor baseline, gives a useful picture of the field. The results show that raw sensor depth is already hard to beat on normal scenes, while learned models and sensors behave very differently on challenging ones. This clearly indicates where the open problems lie, even though the paper does not fully draw this conclusion itself.
Critical weaknesses
- The positioning is not strong enough. The paper motivates the benchmark with a general claim about robotics and autonomous driving, but its own results show that normal scenes are largely solved: raw sensor depth beats all estimation models there, and existing laser-scanned datasets (e.g., ScanNet++, ETH3D) already provide accurate GT for such scenes. The real need for independently measured GT lies in conditions where sensors fail. In my view, the most important of these is outdoor severe weather. In rain, snow, ice, and flooding, the scene geometry itself changes: water accumulates on surfaces, ice forms on roads, water levels rise, snow and mud shift. LiDAR and active depth sensors are severely degraded; for example, wet surfaces such as roofs or vehicles can return few or no points. These conditions are not rare. For vehicles, drones, and construction robots in northern regions, snow and ice cover the environment for much of the year, and the system must perceive these conditions as they occur. Measurements taken before or after the event cannot serve as GT, because the geometry of interest exists only during it. This should be the motivation for the benchmark.
- The scale, especially of the challenging subset, is too small. Each scene has a single point pair, so the benchmark rests on 100 measurements, and the challenging subset has only 24 scenes. No confidence intervals are reported, so many rank differences are likely within noise. The challenging split is also defined by sensor-depth consistency, which partly predetermines findings about the sensor and the completion models that take sensor depth as input. The authors should use a small but sufficient set of normal scenes to confirm that current methods work there, and devote most of the effort to substantially more challenging scenes, particularly transparent and reflective surfaces and severe weather.
Response to the author's rebuttal:
- Their plan is clear, I think with the expansion, it could be a stronger work in future
Questions for authors
Expansion plan (this would most affect my score):
- Can you outline a concrete plan to expand the challenging subset, especially transparent and reflective surfaces and outdoor severe weather (rain, snow, ice, flooding) with in-situ measurement? A small pilot set would strengthen the case considerably.
- Will the dataset and evaluation code be released, and under what license?
4 comments
Q1: Expansion plan and pilot set (anonymous link).
We will substantially expand the challenging subset with the following scene categories:
- Transparent and semi-transparent surfaces (anytime, indoors and outdoors): glass doors and windows, glass curtain walls, shop windows and storefronts, glass partitions and railings, glass staircases and skywalks, bus shelters, greenhouses, display cases and aquariums, glass tables and shelves, glass bottles, cups, and containers, frosted and patterned glass, acrylic and plastic sheets, plastic curtains and films, and translucent containers.
- Reflective surfaces (anytime, indoors and outdoors): mirrors and mirrored walls, polished metal (e.g., elevator doors, stainless-steel kitchenware, equipment, and sculptures), glossy floors and tiles, lacquered furniture, glossy screens and appliances, car bodies and windshields, and mirrored building facades.
- Rain and wet surfaces (on rainy days, especially during the rainy season): wet roads, sidewalks, and crosswalks, wet roofs and vehicles, wet stairs and ramps, rain-covered glass and windshields, wet construction sites, and scenes during active rainfall.
- Standing water and flooding (during and after heavy rainfall): puddles on roads and sidewalks, waterlogged roads, parking lots, and courtyards, flooded low-lying areas and lawns, and water-covered steps.
- Snow (in winter, or in high-latitude and high-altitude regions): snow-covered roads, sidewalks, and stairs, snow-covered vehicles, roofs, and vegetation, snow piles and drifts, partially cleared roads, and scenes during active snowfall.
- Ice (in winter, or in high-latitude and high-altitude regions): icy roads and sidewalks, frozen ponds and lakes, icicles, frost-covered glass and vehicles, and ice-covered railings, stairs, and structures.
- Fog and haze (on foggy mornings and in seasons with frequent fog): roads, parks, and building surroundings under fog or haze, which degrade both visual and active sensing.
We will capture these scenes following our original pipeline, enriched with the within-frame point pairs of varying difficulty and cross-frame point pairs suggested by Reviewer ycfH, with all distances measured in situ, to scale up the benchmark. As a pilot set, we have already collected 6 scenes: 3 with transparent surfaces, 2 with reflective surfaces, and 1 with a puddle, available at this anonymous link. The remaining categories depend on weather conditions and will be captured as conditions permit.
Q2: Release and license of the dataset and code.
Yes. We will publicly release all data, including the RGB-D sequences, camera intrinsics, tape-measured distances, and per-frame annotations, together with the full evaluation code, under the Apache License 2.0.
I am Satisfied
Keep on the good work
Response to reviewer oU9v:
Thank you for your constructive suggestions. We will expand Pumpire as planned.
W1: Positioning of the benchmark and expansion to sensor-failure scenes such as severe outdoor weather.
We thank the reviewer for this thoughtful suggestion.
(1) On normal scenes being largely solved. By proposing a benchmark that directly evaluates point-pair distance and unifies the evaluation of image- and video-level estimation and completion models, our goal is not to find the best ranging method, but to drive the progress of 3D foundation models. Pumpire aims to guide them toward reliable performance in real-world scenes, including normal ones, and ultimately beyond active sensors.
(2) On severe outdoor weather. We share the reviewer's view that severe outdoor weather, such as rain, snow, ice, and flooding, is critical for real-world perception and an important direction for future research. Pumpire takes the first step by focusing on common real-world scenes. To our knowledge, it pioneers point-pair distance evaluation across method families, including the raw sensor.
(3) On expanding to sensor-failure scenes. Collecting such scenes with in-situ measurement is a key next step, with a concrete plan given in Q1. Nevertheless, the current benchmark is already meaningful on its own.
W2: Limited scale, bias of the sensor-based difficulty split, and expansion of challenging scenes.
(1) Scale. We acknowledge that the scale is limited, but it is comparable to widely used benchmarks: Bonn, ETH3D, and DIODE in the MoGe-2 evaluation contain 26, 13, and 20 scenes, respectively, so even our challenging subset (24 scenes) is of similar size, and the full benchmark (100 scenes) is larger. Enlarging the challenging subset is indeed important, and we will pursue it in future versions.
(2) Difficulty split. We recognize that the sensor-based split may bias findings about the sensor and completion models. We therefore regroup all 100 scenes by surface type, classified by Claude Opus 5.5 from RGB images alone, and report RE:
Under this sensor-independent grouping, Findings 2 and 3 still hold. We will add this analysis to future versions of the paper.
(3) Expanding challenging scenes. We fully agree on its importance and will expand transparent, reflective, and severe-weather scenes in future versions (plan in Q1).