- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a benchmark that evaluates embodied exploration by the 3D reconstruction its captured images support. Agents explore indoor scenes from Matterport3D, iGibson, and ProcTHOR on an offline waypoint graph under movement and frame budgets, and their retained RGB frames are evaluated by coverage, pose-free reconstruction with feed-forward models, and novel-view synthesis. Benchmarking prompted VLMs, VLN policies, and scripted baselines, the paper reports a trade-off in which broader exploration improves whole-scene completion but increases error over the observed region, and identifies view overlap, spatial spread, and frame selection as key factors for reconstruction quality.
Critical strengths
-
Problem formulation. Evaluating exploration by the downstream reconstruction of its captured frames is a well-motivated and under-studied interface between embodied navigation and 3D reconstruction.
-
Design. Decoupling capture from reconstruction allows the same capture set to be replayed through multiple reconstruction backends, enabling controlled comparisons across both policies and reconstructors.
-
Analysis. The controlled studies of co-visibility, spatial spread, frame selection along fixed walks, and frame ordering provide useful evidence that acquisition structure, not just frame count, drives reconstruction quality.
Critical weaknesses
-
Scenes. All scenes are indoor, so it is unclear whether the conclusions hold for outdoor or large-scale environments; evaluation on outdoor data such as Wanderland [1] is needed.
-
Interface. The discrete waypoint graph is justified only by simulation speed, yet it handicaps policies trained in continuous environments, as the paper itself acknowledges that rankings reflect the action interface. A continuous-environment setting is needed for the benchmark to measure navigation ability rather than adaptation artifacts.
-
Fairness. VLN policies are trained for efficient instruction following, so their low coverage is expected rather than a meaningful finding, and they also follow a different capture rule from the VLM planners. Comparing them on coverage against policies explicitly prompted to explore is not a fair comparison.
-
Reconstruction. Only feed-forward models are evaluated, and key findings, such as error increasing with more frames along a walk and sensitivity to frame order, are tied to drift in streaming models. Conventional SfM with global bundle adjustment would likely behave differently, so the conclusions may not generalize beyond feed-forward reconstruction.
-
Confound. Observed-region error is computed over a region that grows with exploration, so the central coverage–error trade-off is partly a consequence of the metric definition. Moreover, meter-level completion errors and very low PSNR suggest that many reconstructions fail outright, making policy comparisons in this regime hard to interpret.
-
Insights. Sec. 4.2 reports phenomena without explaining them; for example, it does not analyze why many VLM planners cover less than a random walk beyond attributing it to stalling.
-
Baselines. Although the paper identifies co-visibility and frame selection as key factors, it does not propose a baseline that exploits these findings or offer guidance on how to perform better on the benchmark.
-
Positioning. The claim that exploration and reconstruction are typically evaluated separately overlooks active reconstruction work that evaluates both jointly, such as ANM [2], NARUTO [3], and ActiveSplat [4]. The paper should clarify how its pose-free, offline capture-set formulation differs from these.
[1] Liu, Xinhao, et al. "Wanderland: Geometrically grounded simulation for open-world embodied AI." CVPR. 2026.
[2] Yan, Zike, et al.. "Active neural mapping." CVPR. 2023.
[3] Feng, Ziyue, et al. "NARUTO: Neural active reconstruction from uncertain target observations." CVPR. 2024.
[4] Li, Yuetao, et al. "ActiveSplat: High-fidelity scene reconstruction through active Gaussian splatting." RA-L. 2025.
I actually lean towards a borderline rating of the paper at current state despite all these weaknesses. If all my concerns can be addressed during rebuttal, I'm happy to give a positive rating.
Questions for authors
-
How exactly is a stall defined? In particular, does a ROTATE action, which changes the heading but neither the waypoint nor the retained frame set, count toward termination?
-
What instruction is given to the VLN policies during exploration?