- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of existing multi-view 3D consistency metrics, showing that neural reconstruction-based methods can produce misleading scores even for clearly inconsistent images. Through a controlled benchmark and analysis of different metric components, the authors identify hallucinated geometry as a major source of these failures. They further propose COLMAP-based alternatives that rely on explicit geometric verification and show stronger agreement with human judgments.
Critical strengths
1. An important problem
The paper studies a real weakness in current 3D consistency metrics. Learned reconstruction models can hallucinate geometry, causing unrelated images to receive high consistency scores. This matters because such scores may be used to evaluate or filter 3D data.
2. Useful
SysCON3D tests metrics on controlled cases such as outlier views, mixed scenes, and noise. This makes it easier to see which kinds of inconsistencies a metric can detect, and where it fails.
3. Interpretable alternative
The proposed COLMAP-based metrics rely on explicit geometric evidence, such as successful camera registration and dense scene support. The authors also compare their scores with human rankings and clearly discuss cases where geometric reconstruction can fail.
Critical weaknesses
1. Limited coverage
SysCON3D uses nine Mip-NeRF360 scenes and focuses on five types of inconsistency. The tests include cross-scene mixtures and synthetic Gaussian noise, which expose clear failures, but they may not capture subtler problems in real NVS outputs, such as small changes in object shape or geometry that drifts across viewpoints. It would help to evaluate the benchmark on more scenes and include realistic generation errors.
2. COLMAP may penalize consistent scenes that are hard to reconstruct
The proposed metrics use registration success and dense geometric support as evidence of consistency. However, COLMAP can fail on valid scenes with low texture, repeated patterns, reflective surfaces, or limited viewpoint overlap. In those cases, a low score may reflect reconstruction difficulty rather than true inconsistency. The paper acknowledges this limitation, but should report how often these failures occur and how the scores change across different scene types.
3. The practical conclusion is stronger than the evidence
The paper suggests that COLMAP-based metrics are the most reliable alternative, but their strongest results are reported on a limited set of datasets and ranking tasks. The evidence supports the claim that geometric verification is more interpretable and more robust for the tested cases. It does not yet establish that COLMAP is generally the best evaluator for multiview generation, especially in settings with low texture, large viewpoint changes, dynamic content, or incomplete observations.
4. The benchmark may favor the proposed metric design
The authors use SysCON3D to select and improve neural metrics, and then report the best-performing variant, MASt3R-W-IMQ. This creates a possible evaluation bias: the benchmark may be used both for metric development and for final comparison. The paper should clearly separate development and test sets, or evaluate the final metric on unseen corruption types and scenes. Otherwise, the reported improvement may partly reflect adaptation to the benchmark rather than genuine robustness.
Questions for authors
1. How well does SysCON3D represent real-world failures?
SysCON3D is built from nine Mip-NeRF360 scenes and uses controlled corruptions such as cross-scene mixtures and Gaussian noise. Do the authors have evidence that these tests reflect the kinds of inconsistencies found in real NVS outputs? Could the benchmark be extended to include subtler, naturally occurring errors?
2. How often does COLMAP penalize valid but difficult scenes?
COLMAP may fail on consistent scenes with low texture, repeated patterns, reflective surfaces, or limited viewpoint overlap. How frequently do these cases receive low scores, and can the authors report results broken down by scene type or reconstruction success?
3 comments
Thanks for your detailed rebuttal, I have raised my score.
Thank you. We are glad our responses addressed your concerns.
POST 7, Response to reviewer Hfeh
Thank you for the thoughtful review.
W4. Benchmark used for both development and final comparison.
The reviewer misunderstands.
W1/Q1 (real-world failures; more scenes). We agree that SysCON3D does not represent real NVS failures: it tests a necessary condition that every metric should pass easily. The new SysCON3D-UE (G6, App. A) adds the subtle errors you mention, small shape changes and geometry that drifts across viewpoints, together with moved, appearing or disappearing objects, floaters, ghosting and appearance changes: 702 samples in 4 indoor scenes along 6 camera trajectories. All five 3D backbones keep 90–100% of the injected points; COLMAP keeps isolated injected geometry at about half the rate of real geometry (App. A, Figs. 4–6). For real NVS outputs, G2 reports 13 methods on the 140 DL3DV benchmark scenes and the 9 Mip-NeRF360 scenes, with a new human study.
W2/Q2 (valid but difficult scenes). See G7. We acknowledge that the submission does not quantify false penalties on valid difficult scenes. SysCON3D-UE now measures them: across 169 consistent samples, SIFT registers 72% of the views (29% in two low-texture conference rooms, 94% at textured sites), and ALIKED + LightGlue register 90% (76% and 98%); the remaining failures are mirror floors and untextured surfaces. On real NVS outputs, a learned extractor and matcher reconstruct every sequence that fails with SIFT, while W-GPC's human alignment changes by at most 0.11 (Table 6, App. M.7). SIFT failures come mostly from methods that humans rank low (G7).
W3 (practical conclusion). See G7.
We hope these clarifications address your concerns, and we would be grateful if you would take them into account in your final assessment.