- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper uses experiments across 6 datasets and 14 foundation models to evaluate the effectiveness of Chamfer Distance (CD) as a metric for 3D reconstruction. The authors show that on a synthetic dataset (i.e., LuSNAR), even an incorrect scene evaluated with CD can outperform foundtaional model 3D reconstruction results. Additionally, the authors provide an analytical bound on CD by comparing the best-fit plane of projected 3D ground-truth (GT) points against the 3D GT points.
Critical strengths
- The paper conducts extensive experiments and effectively uses empirical results to demonstrate that CD may not be suitable for evaluating 3D reconstruction results of low-relief terrain.
Critical weaknesses
- The three contributions claimed by the authors may not appear sufficient for publication:
-
The first contribution relies on basic geometry to derive a bound on CD that is relatively straightforward to obtain.
-
The second contribution uses experiments to illustrate this failure mode—showing that an incorrect terrain can outperform predictions relative to the ground truth—and proposes a simple evaluation alternative based on correlating local height variations on a grid after plane removal and testing completeness via nearest-neighbor distances.
-
In the third contribution, the authors primarily summarize analyses of navigation-related measures, statistical resolution, and evaluation-protocol sensitivity without articulating a distinct, novel technical contribution. While these analyses may be informative, their novelty and demonstrated significance may not meet the bar for publication at ICLR.
-
Since LuSNAR is a synthetic dataset generated in Unreal Engine, could repetitive patterns between the donor and target terrains artificially improve the Chamfer Distance?
-
The core findings regarding CD are strictly limited to low-relief terrain. In contrast, foundation models are commonly benchmarked on domains with richer geometry, such as object-centric scenes, buildings, and indoor environments. This narrow focus limits the broader impact of the findings for 3D reconstruction evaluation. Furthermore, the fine-tuning experiments were conducted on only 3 models across 2 scenes.
-
While the abstract narrows the scope to low-relief terrain, the paper's title and the opening sentences of the abstract frame the work as a general study of 3D reconstruction.
-
The paper is somewhat difficult to follow, as the LLM-assisted drafting and editing appear to lack sufficient final polishing. I suggest that the authors carefully revise the manuscript to improve clarity and readability.
Questions for authors
-
Could the authors rephrase their contributions to more clearly delineate their novel technical findings?
-
The paper should be revised so that it does not read as though it targets general 3D reconstruction.
-
Please clarify whether LuSNAR has repeated patterns that could potentially affect the results.