acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Multiview 3D Consistency when 3D Foundation Models Hallucinate

Abstract

Multiview 3D consistency metrics assess whether generated images depict a single coherent, static 3D scene (eg, MEt3R scores random noise images to be near perfectly 3D consistent). We find that neural metrics, including MEt3R, can assign high consistency to inconsistent image sets and disagree substantially with human judgments on novel view synthesis (NVS) outputs. To diagnose these failures, we use SysCON3D, a controlled robustness benchmark that systematically isolates 3D inconsistency. We decompose reconstruction-based neural metrics into backbone, residual, and aggregation components, yielding a parametric family of neural metrics for controlled analysis and metric improvement. Despite our best neural variant being more robust than MEt3R on SysCON3D, crucial failure modes persisted. Our analysis identifies the cause of failure: feed-forward reconstruction backbones, including VGGT, MASt3R, DUSt3R, and Fast3R, hallucinate geometry and cross-view support for unrelated scenes and random noise. Motivated by these limitations, we introduce COLMAP-based metrics that use verified matches, camera registration, dense support, and reconstruction failure as explicit evaluation signals, and achieve the strongest human alignment of all metrics in our experiments.

Primary area:
interpretability and explainable AI
Keywords:
multiview 3D consistency, novel view synthesis evaluation, ground-truth-free metrics, metric robustness, learned 3D reconstruction, hallucinated geometry, geometric verification, COLMAP, Structure-from-Motion, human alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HfehOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of existing multi-view 3D consistency metrics, showing that neural reconstruction-based methods can produce misleading scores even for clearly inconsistent images. Through a controlled benchmark and analysis of different metric components, the authors identify hallucinated geometry as a major source of these failures. They further propose COLMAP-based alternatives that rely on explicit geometric verification and show stronger agreement with human judgments.

Critical strengths

1. An important problem

The paper studies a real weakness in current 3D consistency metrics. Learned reconstruction models can hallucinate geometry, causing unrelated images to receive high consistency scores. This matters because such scores may be used to evaluate or filter 3D data.

2. Useful

SysCON3D tests metrics on controlled cases such as outlier views, mixed scenes, and noise. This makes it easier to see which kinds of inconsistencies a metric can detect, and where it fails.

3. Interpretable alternative

The proposed COLMAP-based metrics rely on explicit geometric evidence, such as successful camera registration and dense scene support. The authors also compare their scores with human rankings and clearly discuss cases where geometric reconstruction can fail.

Critical weaknesses

1. Limited coverage

SysCON3D uses nine Mip-NeRF360 scenes and focuses on five types of inconsistency. The tests include cross-scene mixtures and synthetic Gaussian noise, which expose clear failures, but they may not capture subtler problems in real NVS outputs, such as small changes in object shape or geometry that drifts across viewpoints. It would help to evaluate the benchmark on more scenes and include realistic generation errors.

2. COLMAP may penalize consistent scenes that are hard to reconstruct

The proposed metrics use registration success and dense geometric support as evidence of consistency. However, COLMAP can fail on valid scenes with low texture, repeated patterns, reflective surfaces, or limited viewpoint overlap. In those cases, a low score may reflect reconstruction difficulty rather than true inconsistency. The paper acknowledges this limitation, but should report how often these failures occur and how the scores change across different scene types.

3. The practical conclusion is stronger than the evidence

The paper suggests that COLMAP-based metrics are the most reliable alternative, but their strongest results are reported on a limited set of datasets and ranking tasks. The evidence supports the claim that geometric verification is more interpretable and more robust for the tested cases. It does not yet establish that COLMAP is generally the best evaluator for multiview generation, especially in settings with low texture, large viewpoint changes, dynamic content, or incomplete observations.

4. The benchmark may favor the proposed metric design

The authors use SysCON3D to select and improve neural metrics, and then report the best-performing variant, MASt3R-W-IMQ. This creates a possible evaluation bias: the benchmark may be used both for metric development and for final comparison. The paper should clearly separate development and test sets, or evaluate the final metric on unseen corruption types and scenes. Otherwise, the reported improvement may partly reflect adaptation to the benchmark rather than genuine robustness.

Questions for authors

1. How well does SysCON3D represent real-world failures?

SysCON3D is built from nine Mip-NeRF360 scenes and uses controlled corruptions such as cross-scene mixtures and Gaussian noise. Do the authors have evidence that these tests reflect the kinds of inconsistencies found in real NVS outputs? Could the benchmark be extended to include subtler, naturally occurring errors?

2. How often does COLMAP penalize valid but difficult scenes?

COLMAP may fail on consistent scenes with low texture, repeated patterns, reflective surfaces, or limited viewpoint overlap. How frequently do these cases receive low scores, and can the authors report results broken down by scene type or reconstruction success?

3 comments
AuthorsOpenReview ↗

POST 7, Response to reviewer Hfeh

Thank you for the thoughtful review.

W4. Benchmark used for both development and final comparison.

The reviewer misunderstands.

  1. The selection is in-sample. Yes, that is correct; MASt3R-W-IMQ is the best of 20 variants on SysCON3D, so its 71% win rate there is an in-sample result (Table 2). But this does not affect our conclusions about the COLMAP-based scores: even the selected variant scores Gaussian noise as more consistent than real captures at every tested KK (Sec. 4.2), and it disagrees with humans on real NVS outputs (ρ=−0.34/−0.45/−0.25\rho = -0.34/-0.45/-0.25 on DL3DV; Table 5). Like MEt3R and PRISM-MMD, it ranks methods largely by frame sharpness, favoring sharp but drifting diffusion outputs over consistent but blurrier feed-forward renderings (App. L).
  2. The final comparison is not made on SysCON3D.: We assess the alignment between metric scores and human rankings for NVS sequences generated by video diffusion and feed-forward models. Even though the mipnerf scenes appear in the benchmark, they appear as real images with synthetic corruptions, not as outputs of NVS methods.

W1/Q1 (real-world failures; more scenes). We agree that SysCON3D does not represent real NVS failures: it tests a necessary condition that every metric should pass easily. The new SysCON3D-UE (G6, App. A) adds the subtle errors you mention, small shape changes and geometry that drifts across viewpoints, together with moved, appearing or disappearing objects, floaters, ghosting and appearance changes: 702 samples in 4 indoor scenes along 6 camera trajectories. All five 3D backbones keep 90–100% of the injected points; COLMAP keeps isolated injected geometry at about half the rate of real geometry (App. A, Figs. 4–6). For real NVS outputs, G2 reports 13 methods on the 140 DL3DV benchmark scenes and the 9 Mip-NeRF360 scenes, with a new human study.

W2/Q2 (valid but difficult scenes). See G7. We acknowledge that the submission does not quantify false penalties on valid difficult scenes. SysCON3D-UE now measures them: across 169 consistent samples, SIFT registers 72% of the views (29% in two low-texture conference rooms, 94% at textured sites), and ALIKED + LightGlue register 90% (76% and 98%); the remaining failures are mirror floors and untextured surfaces. On real NVS outputs, a learned extractor and matcher reconstruct every sequence that fails with SIFT, while W-GPC's human alignment changes by at most 0.11 (Table 6, App. M.7). SIFT failures come mostly from methods that humans rank low (G7).

W3 (practical conclusion). See G7.

We hope these clarifications address your concerns, and we would be grateful if you would take them into account in your final assessment.

Thanks for your detailed rebuttal, I have raised my score.

Thank you. We are glad our responses addressed your concerns.

Reviewer_54c1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the reliability of neural multiview 3D consistency metrics under controlled corruptions. It introduces SysCON3D, a benchmark of inconsistent multiview inputs, and decomposes reconstruction-based metrics into backbone, residual, and aggregation components. The analysis shows that learned 3D backbones can hallucinate geometric support on unrelated scenes and noise, leading to misleading consistency scores. The authors subsequently propose COLMAP-based verification metrics and report strong alignment with human rankings of NVS methods.

Critical strengths
  1. The work addresses a consequential failure mode: metrics relying on learned geometry may assign favorable scores to inputs that lack valid multiview support.
  2. The backbone-residual-aggregation decomposition and controlled corruption tests provide useful insights into where neural metrics fail.
  3. The analysis spans several prominent 3D reconstruction models and connects hallucinated correspondences to unreliable feature-based residuals.
Critical weaknesses
  1. W-GPC combines geometric–photometric agreement with angular coverage through a multiplicative formulation. While coverage is relevant for 360° NVS, the paper does not sufficiently justify how this combination reflects actual 3D consistency. Notably, angular coverage alone achieves high human alignment (ρ = 0.93/0.95), close to W-GPC (ρ = 0.98). Further analysis of the contribution and interaction of these components would strengthen the metric's validity.

  2. The human rankings combine 3D consistency, visual realism, and plausibility with weights of 2:1:1. This makes it difficult to determine whether the reported correlations reflect geometric consistency or broader perceptual quality. Since consistency judgments were collected separately, reporting correlations against consistency-only human rankings would provide a more direct validation of the proposed metrics.

  3. SysCON3D effectively identifies failures under controlled corruptions, including cross-scene mixtures and synthetic noise. However, it remains unclear whether the reported improvements generalize to subtler NVS failures, such as object deformation, inconsistent occlusions, and view-dependent geometry. A more targeted evaluation of these failure modes would strengthen the practical relevance of the findings.

  4. The proposed metrics primarily combine existing COLMAP verification signals, making their added value over simpler alternatives unclear. Moreover, the SysCON3D evaluation of W-GPC focuses on selected settings at K=21, whereas neural metrics are evaluated across seven viewpoint counts. More comprehensive robustness comparisons and component ablations, particularly regarding reconstruction failure handling, would better substantiate the proposed formulation.

Questions for authors
  1. Since angular coverage alone achieves high human alignment (ρ=0.93/0.95), could you provide an ablation study or deeper analysis clarifying the exact contribution of the geometric-photometric agreement and justifying the multiplicative formulation?

  2. Could you report the metric correlations against the consistency-only human rankings, rather than the 2:1:1 composite (consistency:realism:plausibility), to more directly validate that the metrics specifically measure 3D consistency?

  3. How well do the proposed metrics generalize to subtler, more realistic NVS artifacts (e.g., object deformation, inconsistent occlusions, view-dependent geometry), compared to the severe synthetic corruptions evaluated in SysCON3D?

  4. Could you evaluate W-GPC across various viewpoint counts (similar to the 7 counts used for neural metrics, rather than just K=21) to better substantiate its robustness and added value over simpler COLMAP-based alternatives?

2 comments
AuthorsOpenReview ↗

POST 6, response to reviewer 54c1

Thank you for the detailed review. Most of your questions are shared with other reviewers and answered in the General Response; here we add the points specific to your review.

Q1 (coverage vs. agreement; product form). See G3. Coverage and W-GPC rank the methods very similarly, and no paired difference is significant, so we do not claim that depth agreement explains the human alignment. We keep the dense term because coverage only records where the cameras are: a generated view can register on a consistent background while an object in it is wrong, and only the dense term sees that (SysCON3D-UE, App. A). It also makes W-GPC less sensitive to the matcher. The product is a design choice.

Q2 (consistency-only rankings). See G4: all main-text correlations now use consistency-only rankings, with 95% intervals (Tables 3–5).

Q3 (subtle artifacts). See G6. SysCON3D tests a necessary condition that every metric should pass easily. The new SysCON3D-UE (App. A) is closer to real NVS artifacts and covers your three examples: object deformation (stretched objects), inconsistent occlusions (objects that move, appear or disappear in some views), and view-dependent geometry (an object that is correct only from the middle input, and edits that switch or drift along the trajectory). The 3D backbones that learned metrics build on keep 90–100% of the injected points, as often as real geometry, because they predict a depth for every pixel whether or not other views support it. COLMAP's verified support keeps an isolated injected object at about half the rate of real geometry, but reconstructs edits that majority of frames share -- this is a feature, not a bug.

Q4 (view counts, simpler alternatives, failure handling). See G5, G3 and G2.

  • View counts. The revision reports K=3,6,9K = 3, 6, 9 on both datasets for 13 methods (Tables 3, 4); W-GPC's alignment varies by at most 0.08 across KK.
  • Simpler COLMAP alternatives. Registration rate, coverage, ICM, ICMallICM_\mathrm{all}, GPCallGPC_\mathrm{all}, density and depth agreement are all reported (Tables 25–30) and compared in G3. Scores that ignore failed views align clearly worse on DL3DV; coverage alone ties W-GPC.
  • Failure handling. Counting views without dense maps as failures changes the alignment by at most 0.03 (Table 32).
  • Controlled tests across view counts. The submission evaluates the COLMAP scores on SysCON3D only at K=21K{=}21; see G5 and G6.

W4: the metrics "primarily combine existing COLMAP verification signals". This is by design. Our contribution is not a new reconstruction algorithm but three things:

  1. Evidence that learned backbones hallucinate geometric support, and that the metrics built on them fail specific inconsistency tests (Sec. 4.2, 4.4, App. A, App. I).
  2. Scores in which a complete reconstruction failure counts as zero support rather than a missing value.
  3. Validation against human judgments: at K=9K{=}9 on DL3DV, W-GPC reaches ρ=0.92\rho = 0.92 against the consistency-only ranking of 13 methods, against −0.25-0.25 for MASt3R-W-IMQ (Table 5).

Keeping each score simple is deliberate: when a score is low, each factor can be inspected.

We hope you will take these answers into account in your final assessment.

AuthorsOpenReview ↗

COLMAP-based scores on the complete SysCON3D protocol

COLMAP-based scores on the complete SysCON3D protocol (fSnK Q3; 54c1 Q4)

As requested, we ran the COLMAP-based scores on the complete SysCON3D protocol: all 7 view counts (K=3,…,21K = 3, \dots, 21), the five corruptions in Table 2, and the full cross-scene mixtures, against the 9 consistent sets for each KK. They are scored exactly as the neural metrics in Table 2 and App. K. COLMAP runs at native resolution, while for the neural metrics, it's dependent on the input resolution required by each

Ordering of the corruption levels. Fig. 2 (last panel) shows the ideal behavior: the score worsens monotonically from consistent captures (L0) through the cross-scene mixtures (L1, L2, L3, full mixture) to Gaussian noise. The table gives the rank each metric assigns to each level, averaged over K=6K = 6–2121 (1 = rated most consistent), and Kendall's τ\tau with the ideal order (App. K, Table 17):

MetricL0L1L2L3Full mixtureGaussian noiseτ
Ideal (Fig. 2)1234561.00
W-GPC2.21.32.54.55.25.20.85
MASt3R-W-IMQ2.52.54.05.06.01.00.26
MEt3R5.85.04.22.72.31.0−0.89

W-GPC ranks random mixtures, full mixtures, and noise last: every full mixture and Gaussian-noise set scores exactly zero at every KK, because COLMAP cannot reconstruct it (hence the tie at 5.2). Both neural metrics rank Gaussian noise as the most consistent level at every KK, and MEt3R inverts the whole order. None of the 20 MASt3R, Fast3R and VGGT variants of Table 2 exceeds τ=0.30\tau = 0.30 (Table 17). W-GPC keeps τ=0.74\tau = 0.74 over K=6K = 6–2121 if its tied zero scores are counted as ties rather than in the ideal order.

Win rates (Table 2's criterion). W-GPC wins 21 of the 31 decided comparisons (68%): L1 1/6, L2 4/6, L3 6/6, patched Gaussian 4/7, Gaussian noise 6/6. MASt3R-W-IMQ wins 25/35 (71%) and MEt3R 8/35 (23%). Four comparisons are undecided: at K=3K = 3, COLMAP reconstructs no consistent set and no set of 4 corruption types, so all their scores are zero, and Cohen's dd is undefined; Table 2's protocol drops such comparisons.

Limitations

  • Sparse views. The consistent sets are fixed views spread around 360° scenes. COLMAP reconstructs none of them at K=3K = 3, 1 of 9 at K=6K = 6, and 8 of 9 at K=21K = 21 due to limited visual overlap.
  • Single outliers. The rejected outlier barely changes W-GPC (our reply to fSnK). In addition, the corrupted sets draw their real views at random, with replacement, so these views are closer together in camera arc space than the consistent views (2–3 times for single-outlier sets at K≥12K \ge 12), and a third of the single-outlier sets repeat a frame. COLMAP reconstructs such sets more easily, which is why W-GPC ranks L1 above L0 and why L2 and patched Gaussian are nearly even.

The conclusions are the same on 512 px images: W-GPC wins 22 of 31 comparisons (71%, the rate of MASt3R-W-IMQ) with τ=0.85\tau = 0.85. We have replaced Table 31 with these results in the updated version.

Reviewer_fSnKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper investigates the reliability of reference-free multiview 3D consistency metrics. It introduces SysCON3D, a controlled benchmark containing cross-scene mixtures and noise corruptions, and analyzes neural metrics through their reconstruction backbones, residual functions, and aggregation strategies. The study shows that learned backbones can produce unsupported geometry and correspondences, misleading downstream consistency scores even after improving aggregation. The authors then introduce COLMAP-based metrics using verified reconstruction support, depth agreement, and viewpoint coverage. Experiments assess corruption robustness and alignment with composite human rankings of novel-view synthesis methods.

Critical strengths
  1. The paper identifies an important reliability issue: learned multiview consistency metrics can assign favorable scores to unrelated images and noise. Controlled experiments across several reconstruction backbones make this finding convincing.

  2. The backbone analysis provides useful diagnostic insights, and the COLMAP-based metrics show promising alignment with human judgments on NVS outputs.

Critical weaknesses
  1. W-GPC computes dense agreement over successfully reconstructed views. Rejected inconsistent views may therefore leave the score unchanged, limiting its ability to penalize partial reconstruction failures.

  2. Human rankings combine consistency, realism, and plausibility. Coverage alone also achieves correlations close to W-GPC (Table 4), leaving the additional geometric discrimination provided by depth agreement unclear.

  3. The proposed COLMAP metrics are evaluated on only selected SysCON3D conditions at K=21, leaving their robustness across the complete benchmark insufficiently established.

Questions for authors
  1. How does W-GPC penalize inconsistent views that are rejected without changing the reconstructed subset or its coverage? How does it compare with ICM_all in this setting?

  2. Could you report consistency-only human alignment with confidence intervals, and clarify the additional benefit of depth agreement over coverage alone?

  3. Could you evaluate the proposed metrics on the complete SysCON3D protocol and include physically consistent but difficult-to-reconstruct scenes to assess false penalties?

4 comments
AuthorsOpenReview ↗

POST 5, response to reviewer fSnK

Thank you for the constructive review. W2/Q2 and W3/Q3 are answered in G3–G5 and G7; we answer W1/Q1 here.

W1/Q1. A rejected view that changes neither the reconstructed subset nor its coverage; comparison with ICM$_\mathrm{all}$.

You are right: in that case W-GPC does not penalize the rejected view. W-GPC penalizes an inconsistent view only through its effects:

  • if the view registers, through depth disagreement or lost dense support (GPC);
  • if it is rejected, through a larger angular gap between the remaining cameras (coverage).

A rejected view that changes neither escapes both. ICMallICM_\mathrm{all} and the registration rate do penalize it: each additional zero-support view enlarges ICMallICM_\mathrm{all}'s denominator ∑i∈TMi\sum_{i\in\mathcal{T}} M_i and lowers the score, and the registration rate ∣R∣/∣T∣|\mathcal{R}|/|\mathcal{T}| drops. This is why we recommend reporting W-GPC together with ICM and the registration rate (Sec. 5, App. E).

Does this blind spot change the method rankings? In our data, it does not, for two reasons.

  1. W-GPC aligns with humans at least as well as ICM$_\mathrm{all}$: higher in four of six settings and tied in two:
DL3DV K=3K=6K=9Mip K=3K=6K=9
W-GPC0.860.840.920.840.800.87
ICM$_\mathrm{all}$0.830.810.920.810.800.84
  1. Counting every view without a dense map as a failure (W-GPC · ∣J∣/∣T∣|\mathcal{J}|/|\mathcal{T}|) changes human alignment by at most 0.03, never significantly (Table 32). In 97% of the reconstructions every registered view has a dense map, so the ignored views are mostly unregistered ones, which already reduce coverage.

A controlled case close to yours. In SysCON3D-UE (G6), an artifact confined to one generated view usually leaves that view registered, so coverage does not change. COLMAP's dense verification still keeps the injected pixels at about half the clean rate, and 74–81% of these lie on the true surface (App. A). The unsupported content thus loses dense support, which lowers GPC.

Revision. FAQ 8 now states this limitation, defines GPC$_\mathrm{all}$, and corrects the submission's claim that failed registration or densification always counts as zero support: this holds only for complete failure and for the attempted-set scores.

W2/Q2. All rank correlations now use consistency-only human rankings with 95% bootstrap intervals (G4; Tables 3–5). G3 covers this.

W3/Q3. G5 reports the COLMAP scores at K=3,6,9K = 3, 6, 9 on real NVS outputs, and G6 adds the controlled benchmark. G7 covers valid but difficult scenes that the submission did not quantify: on 169 consistent SysCON3D-UE samples, SIFT registers 72% of the views, and ALIKED + LightGlue register 90%.

We hope these answers address your concerns, and we would be grateful if you would take them into account in your final assessment.

Thank you for the detailed responses and additional experiments. Most of my concerns have been addressed.

I am rasing my rating to 3.

Thank you for your response. We are glad the rebuttal addressed most of your concerns.

We could not find this point in our submission or in any of the previous reviews, so we are not sure we understand the question correctly. Our paper does not claim an efficiency benefit. Its only runtime statements are that our COLMAP pipeline takes about 5 minutes per sequence (App. M.3) and that COLMAP is slower than the neural metrics (Sec. 6). If you wanted a comparison of the runtime needed by the neural and the COLMAP-based metrics, here are end-to-end times measured directly, per sequence, during our human-study evaluation, from the image files to the final score (all COLMAP stages, or model loading and scoring):

MetricFrames per sequenceMedian time per sequence
COLMAP, SIFT (all COLMAP-based scores from one run)all generated frames + inputs (median 56)4.5 min (5,359 sequences)
COLMAP, ALIKED + LightGlueall generated frames + inputs (median 56)12.5 min (702 sequences; SIFT on the same sequences: 5.0 min)
MEt3R25 generated frames + inputs (28–34)47 s (5,364 sequences)

These runs used a mix of GPUs, so they are not a controlled comparison, and the neural metrics scored fewer frames than COLMAP (App. M). This matters because the neural metrics slow down quadratically with the number of frames: MEt3R and MASt3R-W-IMQ score every pair of frames, so NN frames cost N(N−1)/2N(N-1)/2 backbone evaluations, 561 pairs for 34 frames and 5,995 for 110.

To measure the effect of NN directly rather than estimate it, we are timing all four metrics (COLMAP, MEt3R, MASt3R-W-IMQ) end to end on the same GPU type (one RTX A5000 and 4 CPU cores per run), on the same generated sequences with N=16,32,64N = 16, 32, 64 and 110110 frames. We will post the table as soon as the runs finish.

As promised, here are runtimes measured end-to-end on identical hardware: one RTX A5000 GPU and 4 CPU cores per run, from the image files to the final score. We use DepthSplat outputs for two Mip-NeRF 360 scenes (garden/counter, 768×448), with N=16,32,64N = 16, 32, 64 evenly spaced frames and all 110110 generated frames. COLMAP's time covers feature extraction, matching, mapping, dense stereo, and scoring; a single run yields all COLMAP-based scores. The neural metrics' time includes startup, model loading, and scoring all N(N−1)/2N(N-1)/2 frame pairs.

Minutes (garden / counter)N=16N = 1632326464110110
Frame pairs1204962,0165,995
COLMAP, SIFT1.3 / 0.57.9 / 8.318.4 / 17.435.1 / 32.2
COLMAP, ALIKED + LightGlue4.0 / 4.18.9 / 9.119.6 / 19.037.4 / 34.8
MEt3R0.7 / 0.71.5 / 1.65.1 / 5.214.5 / 14.7
MASt3R-W-IMQ0.8 / 0.82.1 / 2.17.5 / 7.721.7 / 22.2

At N=16N = 16, the frames are far apart, and COLMAP with SIFT registers only 6 and 2 of them, so the run ends early; with ALIKED + LightGlue, it registers all 16 frames in both scenes.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Summary of paper revision

Summary of the revision

We apologize for the late response. We used the discussion period to substantially expand our experiments, and running them took longer than we expected. The revised PDF is uploaded; new or changed text and tables are highlighted in yellow.

A short video of the whole paper. Because the revision adds many results, we have shared a video of under three minutes in the supplementary material, to ease the work of the AC and reviewers. It walks through the main findings: how learned 3D backbones hallucinate geometry on SysCON3D and SysCON3D-UE, how COLMAP rejects it, the new human study, and why the neural metrics reward sharp frames.

Main results

  1. A much larger evaluation, with a new human study. It covers 13 NVS methods instead of 8 and the 140 DL3DV benchmark scenes instead of 24, plus the 9 Mip-NeRF360 scenes, at K=3,6,9K = 3, 6, 9: 5,811 generated sequences. A new human study ranks all 13 methods by 3D consistency alone (G2, G4). To our knowledge, this is the largest evaluation of 3D-consistency metrics against human judgments of NVS outputs.

  2. The neural metrics we evaluate do not agree with humans. On DL3DV, MEt3R, PRISM-MMD and our best variant MASt3R-W-IMQ all correlate negatively with the human consistency rankings at every KK (ρ\rho from −0.68 to −0.25). On Mip-NeRF360, none of them are significant, while W-GPC is significant on the same 9 scenes (G2, G5).

  3. Our COLMAP-based metrics do. W-GPC reaches ρ=0.80\rho = 0.80–0.920.92 against the same rankings, significant in all six settings after Holm correction. Together with its angular-coverage component, the COLMAP-based scores are the most human-aligned of all the metrics we evaluate. W-GPC's agreement with humans barely changes when SIFT is replaced by a learned feature extractor and matcher (G7), or when views without dense maps are counted as failures (G3).

  4. The neural metrics reward sharpness, not consistency. Diffusion-based methods produce sharp frames that drift across views; feed-forward methods render every frame from one 3D representation, so their frames are consistent but blurrier. Humans and W-GPC rank the feed-forward methods higher on average. MEt3R and MASt3R-W-IMQ do the opposite: on DL3DV, they rank all seven diffusion-based methods above all six feed-forward methods, at every KK. All three neural rankings follow the ranking by frame sharpness (ρ=0.65\rho = 0.65–0.890.89 at K=9K{=}9), while the human and W-GPC rankings do not (ρ=−0.29\rho = -0.29 and −0.35-0.35). The backbones and features behind these metrics are trained on sharp images and appear to treat sharp detail as evidence of consistency. COLMAP instead checks whether the views agree geometrically, however sharp they are (Sec. 4.3, App. L).

  5. Why this matters. Learned 3D backbones can produce geometry and cross-view support for images that share no underlying 3D scene, and metrics built on them inherit such failure modes. We believe this is among the most important findings to date for the safe evaluation of NVS and 3D reconstruction models: an evaluator must reject geometry that the images do not support. Explicit geometric verification does; the evaluated foundation models do not.

  6. Measuring hallucination directly, with 3D ground truth. To our knowledge, we are the first to analyze and ablate this hallucination systematically across 3D backbones (Sec. 4.4, App. I). The revision adds SysCON3D-UE (App. A, at the front of the appendix). SysCON3D tests a necessary condition that every metric should pass easily; SysCON3D-UE brings the evaluation closer to the artifacts we see in the outputs of the 13 NVS methods. It has 702 samples rendered with Unreal Engine 5 in 4 indoor scenes with artifacts typical of video-diffusion and regression models: deformed, moved, appearing or disappearing objects, view-dependent depth errors, floaters, ghosting, and appearance and exposure changes. Because the 3D scene geometry is known, we can count the unsupported ("ghost") 3D points that each method keeps. All five 3D backbones, including the latest VGGT-Ω, keep 90–100% of them, as often as real geometry. This exposes a direct mechanism behind the neural metrics' disagreement with humans: metrics built on these backbones get no geometric evidence against inconsistent content, so sharp frames that drift across views can still score as consistent. COLMAP's geometric verification keeps isolated artifacts at about half the rate of real geometry (G6).

The shared questions are answered below; the individual responses follow.

AuthorsOpenReview ↗

General Response to all reviewers

We thank the reviewers for their feedback. We are glad they found that the paper addresses "a consequential failure mode" (54c1) and "an important reliability issue" (fSnK). They also found that controlled experiments across several backbones make the hallucination finding "convincing" (fSnK), and that the backbone–residual–aggregation decomposition gives "useful insights into where neural metrics fail" (54c1). Finally, they called the COLMAP-based metrics an "interpretable alternative" with clearly discussed failure cases (Hfeh).

Shared questions are answered below, and individual responses follow. In the revised PDF, new or changed text and tables are highlighted in yellow.

Planned changeRaised byStatus
13 NVS methods, 140 DL3DV scenes and a new human study (G2)Hfehin the revised PDF: Sec. 4.3, 5; Tables 3–5; App. G, H, M.5–M.6
Neural metrics follow frame sharpness, not consistencynew findingin the revised PDF: Sec. 4.3; App. L, Table 18
COLMAP component summary and failure-handling ablation (G3)fSnK, 54c1in the revised PDF: App. M.6, Tables 25–30, 32
Consistency-only human rankings with uncertainty (G4)fSnK, 54c1in the revised PDF: Tables 3–5
COLMAP metrics across view counts and under controlled corruptions (G5)fSnK, 54c1real NVS outputs: Tables 3–5; controlled: App. A (G6)
SysCON3D-UE: realistic NVS artifacts with 3D ground truth (G6)54c1, Hfeh, fSnKin the revised PDF: App. A (Tables 7–9, Figs. 4–6)
COLMAP with a learned feature extractor and matcher, ALIKED + LightGlue (G7)Hfeh, fSnKin the revised PDF: Table 6, App. M.7, FAQ 19
AuthorsOpenReview ↗

POST 1, General Response (1/4)

Our evaluation has two stages with different roles; a paragraph at the start of Sec. 3 will make this explicit in the final version.

  1. SysCON3D is a stress test of necessary conditions. It checks whether a metric rates image sets that do not depict a real 3D scene (cross-scene mixtures, noise) as less consistent than real multiview captures. Every reliable metric should pass it easily: failing it reveals a basic reliability problem, and passing it is not sufficient (App. C, FAQ 1–2). We use it to diagnose metrics (Sec. 4.2, 4.4). The new SysCON3D-UE (App. A, G6) is closer to real NVS artifacts.
  2. Validation is method-level agreement with human rankings of real NVS outputs. The revised study covers 13 methods that generate 360° trajectories on the 140 DL3DV benchmark scenes and the 9 Mip-NeRF360 scenes (Sec. 4.3, 5; Tables 3–5). The DL3DV scenes are not in SysCON3D. The Mip-NeRF360 scenes are, but the evaluated sequences are generated by video diffusion or feed-forward 3D models, not ground-truth views with synthetic corruptions.

G2. More methods, more scenes, and a new human study (Hfeh W1, W3)

We completed the expanded evaluation: 13 NVS methods instead of 8 (we added AnySplat, ReSplat, GEN3C, FrameCrafter, and WorldMirror, adapted to 360° orbits as described in App. H), and the 140 official DL3DV benchmark scenes instead of 24 (118 new) plus the 9 Mip-NeRF360 scenes, at K=3,6,9K = 3, 6, 9. That is 5,811 generated sequences, evaluated with the same metrics as in the submission. A new human study ranks all 13 methods on both datasets (App. G): 316 DL3DV and 222 Mip-NeRF360 comparisons, ranked per dataset and KK by a Bradley–Terry fit to the consistency votes (G4).

Spearman ρ\rho with the consistency-only human ranking of 13 methods (Table 5; * = significant after Holm correction):

MetricDL3DV K=3K=6K=9Mip K=3K=6K=9
W-GPC0.86*0.84*0.92*0.84*0.80*0.87*
Coverage0.91*0.87*0.95*0.81*0.80*0.89*
ICM0.660.640.70*0.79*0.81*0.73*
MASt3R-W-IMQ−0.34−0.45−0.250.100.340.46
MEt3R−0.55−0.68−0.59−0.040.250.21
PRISM-MMD−0.52−0.49−0.27−0.23−0.11−0.20

W-GPC is significant in all six settings, unlike any neural metric; on DL3DV, all three neural metrics show negative correlations. Table 5 gives 95% bootstrap intervals over comparisons and scenes (e.g., W-GPC on DL3DV at K=9K{=}9: [0.80, 0.95]), and the human rankings are stable (split-half reliability 0.83–0.95, Spearman–Brown corrected). App. M.5 states the evaluation protocol and App. M.8 lists all changes from the submission. One affects a submitted method: we corrected a coordinate-frame error in our Difix3D adaptation and regenerated its outputs, so its scores and rank differ from the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.