acceptodds
Withdrawn from ICLR 2027

What a 3D Reconstruction Benchmark Can and Cannot Tell About Geometric Foundation Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Benchmarks for 3D geometric foundation models rank models by surface metrics such as Chamfer distance, without testing whether those metrics can tell a reconstruction of the right scene from one of the wrong scene. This paper tests it for fourteen models on six domains by scoring two substitutes through the unmodified pipeline: another scene's ground truth, planted on the target's cameras, and the target's own ground truth flattened onto a plane. Where the ground has relief, the wrong scene is rejected on 98% of cells. On low-relief ground, rendered lunar terrain and rover imagery of Mount Etna and Saharan sand alike, the flattened ground truth outscores every model on all 21 windows, and on about half of them even when every submission receives the same similarity fit. A closed-form bound explains why: a plane's Chamfer cannot exceed the target's mean deviation from its own plane, so the plane wins wherever the best model errs by more. The wrong scene also outscores every model on most low-relief cells, but on the real rover data only through placement, which the shared similarity fit removes. The ground-level tables here and those the field publishes are too small to separate their leading models once every pair is tested. The controls and the evaluation code will be released, with a checklist for benchmarks on low-relief terrain.

Primary area:
datasets and benchmarks
Keywords:
3D reconstruction, benchmark validity, evaluation methodology, Chamfer distance, geometric foundation models, negative controls, planetary terrain, statistical resolution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_8vqZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper uses experiments across 6 datasets and 14 foundation models to evaluate the effectiveness of Chamfer Distance (CD) as a metric for 3D reconstruction. The authors show that on a synthetic dataset (i.e., LuSNAR), even an incorrect scene evaluated with CD can outperform foundtaional model 3D reconstruction results. Additionally, the authors provide an analytical bound on CD by comparing the best-fit plane of projected 3D ground-truth (GT) points against the 3D GT points.

Critical strengths
  1. The paper conducts extensive experiments and effectively uses empirical results to demonstrate that CD may not be suitable for evaluating 3D reconstruction results of low-relief terrain.
Critical weaknesses
  1. The three contributions claimed by the authors may not appear sufficient for publication:
  • The first contribution relies on basic geometry to derive a bound on CD that is relatively straightforward to obtain.

  • The second contribution uses experiments to illustrate this failure mode—showing that an incorrect terrain can outperform predictions relative to the ground truth—and proposes a simple evaluation alternative based on correlating local height variations on a grid after plane removal and testing completeness via nearest-neighbor distances.

  • In the third contribution, the authors primarily summarize analyses of navigation-related measures, statistical resolution, and evaluation-protocol sensitivity without articulating a distinct, novel technical contribution. While these analyses may be informative, their novelty and demonstrated significance may not meet the bar for publication at ICLR.

  1. Since LuSNAR is a synthetic dataset generated in Unreal Engine, could repetitive patterns between the donor and target terrains artificially improve the Chamfer Distance?

  2. The core findings regarding CD are strictly limited to low-relief terrain. In contrast, foundation models are commonly benchmarked on domains with richer geometry, such as object-centric scenes, buildings, and indoor environments. This narrow focus limits the broader impact of the findings for 3D reconstruction evaluation. Furthermore, the fine-tuning experiments were conducted on only 3 models across 2 scenes.

  3. While the abstract narrows the scope to low-relief terrain, the paper's title and the opening sentences of the abstract frame the work as a general study of 3D reconstruction.

  4. The paper is somewhat difficult to follow, as the LLM-assisted drafting and editing appear to lack sufficient final polishing. I suggest that the authors carefully revise the manuscript to improve clarity and readability.

Questions for authors
  1. Could the authors rephrase their contributions to more clearly delineate their novel technical findings?

  2. The paper should be revised so that it does not read as though it targets general 3D reconstruction.

  3. Please clarify whether LuSNAR has repeated patterns that could potentially affect the results.

Reviewer_RC6jOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits what standard 3D reconstruction benchmarks can establish about geometric foundation models, particularly on low-relief terrain. It evaluates fourteen models across six domains using two diagnostic substitutes: another scene’s ground truth and the target’s ground truth projected onto its best-fit plane. Both can outperform learned reconstructions under conventional surface metrics. A simple analytical bound characterizes when the flattened target must beat a model under Chamfer distance. Additional experiments separate placement errors from surface errors, examine agreement with navigation outcomes, and assess statistical resolution and sensitivity to evaluation protocols. The paper also introduces relief correlation and recommends reporting it alongside completeness, or combining their ranks.

Critical strengths
  1. A compelling empirical characterization of the low-relief regime. The flattened target beats every model on all 21 low-relief audit windows, while the wrong-scene substitute wins on 24 of 31 control cells. The combination of an explicit error bound, rendered and real domains, additional windows, and the POLAR replication makes this substantially more informative than an isolated counterexample.
  2. Careful diagnosis of alternative explanations and statistical limitations. The model-pose controls and common geometry-alignment experiments help distinguish pose advantages from surface differences. Crucially, the paper reports that wrong-scene wins disappear on AMEDS and Erfoud after the common similarity fit. The statistical analysis also distinguishes unresolved comparisons from equivalence and explicitly limits the attainable-significance argument to the chosen testing procedure. These qualifications, together with the protocol sensitivity experiments, make the audit practically useful and more credible.
Critical weaknesses
  1. The central interpretation is not sufficiently established by the control experiments. A flattened target having lower Chamfer distance than a reconstruction establishes that the reconstruction has greater aggregate geometric error than that reference. It does not establish that Chamfer incorrectly orders geometric fidelity or provides no useful information about differences between models. Section 8 and Appendix K appropriately acknowledge this as negative skill against a reference baseline. However, Section 7 recommends withdrawing the surface metric when a control wins, and Table L.1 states that the metric then “orders nothing.” This recommendation requires additional justification, particularly because Chamfer retains substantial agreement with navigation coverage on AMEDS and Erfoud. The distinction between poor model performance and inadequate measurement determines what scientific contribution the audit establishes.
  2. The evidence for the recommended composite ranking remains exploratory. Relief correlation intentionally ignores planar tilt, positive vertical scaling, and missing coverage, although these properties can affect navigation. Combining it with completeness improves coverage agreement, but coverage itself ignores heights. The functional outcomes, partial correlations, held-out-domain analyses, and extension experiments provide useful supporting evidence. Nevertheless, Appendix E identifies the completeness-composite comparisons as untested point estimates, and none of the eleven metric comparisons in Appendix G survives Holm correction. These results support reporting complementary diagnostics more strongly than adopting the composite as a primary benchmark ranking.
Questions for authors
  1. What independent criterion of scene-specific geometric fidelity demonstrates that the existing rankings become misleading when a control wins? How would you distinguish that finding from a distance metric correctly preferring a geometrically closer reference, and how would this distinction change the metric-withdrawal recommendation?
  2. Could you provide paired uncertainty estimates for the proposed relief-correlation-plus-completeness composite against completeness alone and Chamfer on height-sensitive outcomes such as traversability and safe paths? The analysis should respect scene clustering and the single-scene structure of POLAR.
Reviewer_BavLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether surface metrics used in 3D reconstruction benchmarks can distinguish a correct reconstruction from an incorrect one. They run two controls: A wrong scene's ground truth planted on the target's cameras, and the target's own ground truth flattened onto its best-fit plane, evaluating 14 feed-forward geometric foundation models across 6 domains. The authors derive a closed-form bound showing that the flattened target's Chamfer score cannot exceed the target's mean deviation from its plane. The author propose a new metric, relief correlation, rejects both controls while matching Chamfer's navigation agreement.

Critical strengths
  1. The problem the paper raises is important. The authors audit the metric validity of 3D geometric foundation-model benchmarks.

  2. Equation 2 is a simple closed-form bound derived purely from orthogonal projection. The separation of the two failure modes (relief deviation vs. placement) via the shared similarity-fit variant is methodologically careful and convincing.

Critical weaknesses
  1. The proposed solution creates a dimensionality mismatch for 3D foundation models. The paper critiques benchmarks for general 3D models but proposes "relief correlation," which is strictly a 2.5D height-field metric. While suitable for planetary rovers, adopting a 2.5D metric to evaluate general 3D foundation models fragments the evaluation landscape. Benchmarks would have to awkwardly switch between Chamfer for high-relief/objects and Relief Correlation for low-relief terrain, as the paper does not provide a unified 3D metric that solves the fundamental issue.

  2. The scope is limited to low-relief terrain. The core failure is demonstrated only on three low-relief domains with 14 scenes total.

Questions for authors
  1. Given that per-view depth AbsRel rejects almost all wrong-scene cases, does this mean depth metrics are inherently more robust to the low-relief failure than Chamfer.

  2. Since relief correlation is a height-field metric, how do you envision a unified benchmark for general 3D foundation models? Should future benchmarks dynamically switch between Chamfer and relief correlation based on a dev(h_t) threshold, and if so, how would we rank models across a diverse dataset containing both urban objects and flat terrain?

Reviewer_QDCuOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether standard 3D reconstruction metrics differentiate meaningful reconstructions from incorrect alternatives. The authors assess fourteen geometric foundation models across six domains using two controls. The first control is the ground truth of another scene placed within the camera geometry of the target. The second control is the ground truth of the target projected onto its least-squares plane. The paper formulates a bound on the Chamfer distance of the flattened target and demonstrates that this control outperforms all evaluated models on 21 low-relief windows.

Critical strengths
  1. The paper addresses a significant question regarding the validity of geometric reconstruction benchmarks. The negative controls offer a practical method to evaluate what existing metrics actually measure. Integrating these experiments with a closed-form Chamfer bound connects the observed failures to the geometric properties of the target.

  2. The experimental evaluation is extensive. It covers fourteen models, six domains, alternative alignment protocols, and multiple evaluation metrics. I particularly appreciate Section 6, which evaluates the statistical resolution of model rankings and their sensitivity to evaluation choices. These findings are valuable beyond the specific context of low-relief reconstruction.

Critical weaknesses
  1. Chamfer distance measures geometric proximity, not whether a reconstruction originates from the correct scene. If an incorrect surface is geometrically closer to the ground truth than the prediction of a model, assigning that surface a better score is not necessarily a metric failure. It may instead indicate insufficient reconstruction accuracy. I expect the authors to address that in their rebuttal. To be clearer, this issue is particularly relevant to the flattened control, which utilizes the exact ground truth of the target prior to removing the relief. Equation (2) correctly characterizes the Chamfer distance for this control, but it primarily establishes when the models perform worse than this specific planar approximation.

  2. The experiments show that in 24 of 31 cases of low-relief cells under LOCAL placement, the proposed method is favourable, but only 4 of 31 after a shared similarity fit, with no remaining advantage on the real rover domains. Although the authors acknowledge these qualifications, the recommendation in Section 7 to withdraw Chamfer on failed cells seems stronger than what the results establish. I would distinguish more carefully between insufficient reconstruction accuracy, alignment sensitivity, and an intrinsic limitation of the metric.

  3. The proposed relief correlation metric possesses limitations that require further examination. By correlating residual height fields after removing their best-fit planes, it becomes invariant to positive scaling of the relief. The authors acknowledge this property, but it creates a potentially important issue. A prediction preserving the correct relief pattern at an arbitrarily small nonzero amplitude can achieve perfect correlation, while an exactly flat prediction receives zero by convention. The numerical threshold used in the implementation does not resolve this conceptual limitation.

  4. Finally, the 51-page appendix contains substantial useful analysis, but several discussions of the controls and alternative metrics are repeated across sections. Some consolidation, particularly of Appendices D, F, J, and the repeated checklist in L, would improve readability without compromising reproducibility or completeness of the paper.

Questions for authors

Please address my comments above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.