acceptodds
Under review as a conference paper at ICLR 2027

Confidence Is Not Coverage: Budgeted Confidence Ranking Can Fail Under Fixed Reconstruction Budgets

Abstract

Feed-forward multi-view reconstruction models produce dense 3D predictions with per-point confidence, but downstream representations typically retain only a fixed number of points or primitives. A simple strategy is therefore to rank predictions by confidence before sampling. We show that this seemingly natural use of confidence can be mismatched to fixed-budget reconstruction: because confidence scores points independently, ranking can remove all candidates from parts of the scene before a spatial sampler is ever applied. We isolate the effect of confidence ranking with a matched pool test that compares confidence-ranked and uniformly sampled candidate pools of identical size under the same sampler, budget, and initialization. We also derive a coverage ceiling that upper-bounds the reconstruction quality attainable by any sampler that selects from a given pool without moving its points, allowing irreversible support loss to be detected from reference geometry before sampling. Across multiple reconstruction models and benchmarks, confidence-ranked pools can provide substantially less scene support than equal-sized random pools. Mixing ranked and random candidates restores most of the lost performance. Our results show that confidence remains useful for rejecting unreliable predictions, but is not by itself a suitable allocation rule for a tight scene-level budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D reconstruction, feed-forward 3D reconstruction, 3D foundation models, uncertainty estimation, point cloud sampling, scene coverage

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_ciBaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether confidence ranking is an appropriate way to allocate a fixed reconstruction budget. It distinguishes confidence-based filtering from subsequent budget allocation and introduces a matched-pool experiment: confidence-ranked and uniformly sampled candidate pools have identical sizes and pass through the same sampler, output budget, and initialization. The paper also derives a coverage ceiling, U(C)=2r_C/(1+r_C), where r_C measures the reference surface supported by a candidate pool. This bounds the F-score achievable by any coordinate-preserving subset under the specified evaluation conditions. Experiments show substantial support loss from narrow confidence-ranked pools across several reconstruction backbones and datasets. Mixed and confidence-proportional pools recover most of the observed loss. The paper also identifies conditions where the advantage diminishes or reverses.

Critical strengths
  1. The matched-pool design isolates the proposed failure mechanism convincingly. Holding candidate count, sampler, output budget, and initialization fixed makes the comparison substantially more informative than a conventional confidence-threshold sweep. On DTU, the narrow ranked pool achieves an F-score of 23.52, compared with 77.62 for the equal-sized random pool. Its coverage ceiling is only 24.23, demonstrating that most of the deficit is already imposed by pool membership. This provides direct evidence that changing the downstream coordinate-preserving sampler cannot recover the lost performance.
  2. The analysis includes meaningful controls and explicitly characterizes the limits of the finding. The paper examines multiple backbones, pool widths, retained fractions, and random seeds, with scene-level paired uncertainty estimates. It separately demonstrates that confidence filtering helps on raw predictions, reports the ranking reversal at wider BlendedMVS retention, and includes the low-reach VGGT-Omega counterexample. Mixed and confidence-proportional pools provide constructive checks. These results support a qualified conclusion about allocation under tight budgets rather than a blanket recommendation to discard confidence.
Critical weaknesses
  1. The practical impact beyond the controlled selection setting is less established. Appendix B notes that the released VGGT COLMAP exporter already combines confidence thresholding with uniform budgeted sampling, which largely follows the paper’s recommendation. The motivating examples involving per-view opacity selection or pose estimation also differ from the specific global confidence-ranked-pool protocol studied here. Consequently, it remains unclear which established reconstruction pipelines would materially benefit from changing their current allocation rule. The downstream registration experiment provides limited additional evidence: success is defined relative to a transform fitted to ALL, so it measures agreement with that reference rather than independent registration accuracy. Likewise, the coverage ceiling does not directly establish consequences for pipelines that subsequently move points, refine geometry, or optimize rendering primitives. The authors acknowledge these restrictions, but they limit the demonstrated application-level significance of the otherwise convincing selection result.
Questions for authors
  1. Which existing reconstruction pipelines use the specific budgeted confidence-ranking rule examined here? Can the observed failure be connected to a concrete pipeline configuration, particularly given that the released VGGT exporter already uses thresholding followed by uniform sampling?
  2. Can the registration comparison use a reference transform independent of the ALL point set, or report absolute geometric accuracy alongside agreement with ALL? This would help determine whether the observed agreement advantage translates into better registration.
  3. How sensitive are the ranked–random contrast and the retention crossover to the initial eligibility threshold and evaluation tolerance? This would help establish how readily the practical recommendations transfer beyond the selected settings.
2 comments

The failure connects to one concrete released configuration, WorldMirror's optional global cap. The released VGGT exporter already thresholds and then draws uniformly, as we recommend. The closest released match to global confidence ranking is WorldMirror's cap, off by default. We now run it end to end, at shipped settings with voxel merging on and no optimization step. Our uniform selection renders held-out views better than ranked selection by +3.27 dB (95% range 2.89 to 3.66, 21 DTU scenes). With nothing fitted per scene, the splats score 11.73 dB (ranked) and 15.00 dB (uniform), so the finding is the paired difference. The splats come from the remaining views, though WorldMirror's camera pass sees the held-out images, and these are our development scenes. After merging, the ranked set keeps 1.24M Gaussians on average against 2.35M for uniform. At equal count, uniform still leads by +1.25 dB (0.87 to 1.61, positive in 18 of 21 scenes). On point centers with merging off, uniform also leads by +44.27 points at tight budgets. At the shipped cap, which keeps about half of the points, it leads by +2.29 (1.01 to 3.52; Table 28, Appendix J.6).

The random pool still leads after densification. You asked about pipelines that "subsequently move points, refine geometry, or optimize rendering primitives", and we ran one: a fuller Gaussian fit of 30,000 steps with densification and equal final Gaussian counts, on 28 scenes with fixed cameras. Random leads by +1.20 dB PSNR (95% range 0.67 to 1.72), and DTU accounts for the gain. The seven BlendedMVS scenes alone give +0.29 dB, from -0.93 to +1.53, too close to call (Appendix J.2). The VGGT-Omega counterexample you credited came from an error in one of our scripts, and with that corrected random leads there too (general comment, Appendix E). Registration measured without the full eligible set favors random on disturbed DTU. We measure the errors against a camera-derived reference, which we fit from predicted cameras and benchmark calibration and which does not use ALL. Under disturbance on DTU, rotation error is 1.05 degrees for random, 1.06 for ALL and 4.95 for ranked, and translation error shows the same pattern. Our fixed success test of 2 degrees and 1% of radius gives 0% for every DTU arm, including ALL, so on DTU the error values are what separate the arms. On BlendedMVS, disturbed success is higher for random and undisturbed success is higher for ranked (Table 27, Appendix J.5).

Random's lead with a narrow pool excludes zero in 22 of 24 eligibility and tolerance settings. A narrow pool has four candidates per output point. On DTU the random pool leads at all six eligibility cuts we tried, and at the main cut the ceiling excludes recovery at all three tolerances (Appendix J.1). BlendedMVS is more sensitive to tolerance. At the main cut the ceiling rules out 28 of 28 conditions at native and double tolerance and 8 of 28 at half, where random's narrow-pool lead is +10.10 (95% range 3.94 to 19.29). Both exceptions are BlendedMVS at half tolerance, one with the confidence cut at 1 and one with percentile-mask eligibility, and in both the interval spans zero.

The retention crossover, where the random-minus-ranked scene mean first becomes negative, moves with both settings. On BlendedMVS at the main cut it falls at 10% retention at half tolerance, 50% at native and 75% at double. These are scene means, and at native tolerance the interval at 25% retention includes zero. A higher confidence cut also moves it up, and on DTU ranking first overtakes random at 50% retention only with the cut at 1. So we tie the advice to tight budgets, where random's narrow-pool lead stays above zero in 22 of these 24 settings, rather than to any one crossover point.

AuthorsOpenReview ↗

Follow-up (18 Nov): a released configuration

Thank you again for your review. This follows up our 15 Nov reply with a new result. WorldMirror's optional confidence cap, off by default, is a concrete released configuration of the kind you asked about. In a new test where held-out images never reach the network, uniform selection gives higher held-out PSNR than confidence-ranked selection on all ten ETH3D scenes, at the same Gaussian count. We would be glad to hear whether this answers your question.

Reviewer_pxqkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Feed-forward multi-view reconstruction models (e.g. VGGT) emit dense 3D points with per-point confidence, but downstream representations keep only a fixed budget of points/primitives. A natural strategy is to rank by confidence and then sample. The paper shows this can be mismatched to a fixed-budget reconstruction: because confidence scores points independently, ranking can eliminate all candidates from parts of the scene before any spatial sampler runs. Contributions: (1) a matched pool test that isolates the effect by comparing a confidence-ranked pool and an equal-size uniform-random pool through the same farthest-point sampler, budget, and initialization — on DTU at 4×K pool the ranked pool reaches F-score 23.52 vs 77.62 for the random pool; (2) a coverage ceiling U(C), a simple bound (derived from reference geometry before sampling) on the F-score any coordinate-preserving selection from a pool can achieve, used to prove irreversible support loss (rules out recovery in 84/84 VGGT, 78/84 π3, 80/84 DA3 DTU conditions); (3) evidence the effect generalizes across backbones (VGGT, π3, DA3, DUSt3R, AMB3R), datasets (DTU, BlendedMVS, held-out ETH3D), and an ICP alignment task, and that mixing ranked+random (50/50) recovers 99% of the loss. Takeaway: confidence is a good filter but not a stand-alone allocation rule under a tight scene-level budget.

Critical strengths
  1. Important, well-isolated practical insight. The matched-pool test convincingly demonstrates that confidence ranking (not the sampler) discards whole regions of scene support under tight budgets — directly relevant to anyone pruning VGGT/feed-forward predictions to a point budget. This is useful, corrective guidance.
  2. The coverage ceiling is a genuinely useful contribution. A computable pre-sampling bound that detects irreversible support loss is both theoretically clean and practically actionable, and the broad multi-backbone / multi-dataset / alignment-task validation (including a VGGT-free backbone, DA3) gives the finding real credibility.
Critical weaknesses
  1. The central message is somewhat intuitive and the novelty is moderate. That a pointwise score does not guarantee spatial coverage, and that under aggressive pruning a score-ranked set can fall below a random one, is already established in active learning / coreset literature the authors themselves cite (Sener & Savarese 2018; Ayed & Hayou 2023; Zheng et al. 2023). The paper's advance is to demonstrate it specifically for 3D-reconstruction confidence + FPS and to supply the ceiling; the conceptual "aha" is modest rather than surprising.
  2. Narrow, conditional practical scope. The dramatic effect is tied to very tight pools (4×K keeps only 2.18% of eligible points) and the deterministic top-K → FPS pipeline. At wider pools the result reverses (ranking's higher precision wins on BlendedMVS at 25%/75% retention), and systems that already draw primitives proportional to a score (Mini-Splatting, Taming-3DGS — shown in Appendix E to recover ~97% of the gap) are largely unaffected. So the "bug" primarily concerns a specific subset of pipelines, which bounds the real-world impact of the recommendation.
Questions for authors
  1. Since methods that draw primitives proportional to a score (Mini-Splatting, Taming-3DGS) already recover ~97% of the gap, which currently deployed feed-forward-reconstruction pipelines are actually "doing it wrong" with a deterministic confidence-ranked top-K? Could you clarify the practical population the finding most affects?
  2. Eligibility uses the VGGT-specific threshold conf > 2.0 (from Hillemann et al.). How sensitive are the matched contrasts to this threshold, and do they survive if eligibility is defined solely by the model's own validity mask?
  3. The ceiling assumes a fixed tolerance τ (5 mm on DTU, 1% median depth on BlendedMVS). A short sensitivity analysis over τ would strengthen the claim that the "before sampling" diagnostic is robust, not threshold-dependent.
  4. Beyond "tight budgets," can you give a concrete decision rule (e.g., a pool-width or coverage-ceiling threshold) for when to prefer random/mixed over ranking? This would make the practical guidance sharper.
2 comments

Our rule is for builders who set or tighten a point budget, or who turn on an optional ranking cap such as WorldMirror's. We ran that cap end to end, at shipped settings with its voxel merging on and no optimization step. Our uniform selection then renders held-out views better than ranked selection by +3.27 dB (95% range 2.89 to 3.66, 21 DTU scenes), and by +1.25 dB (0.87 to 1.61, 18 of 21 scenes) when we thin it to the ranked Gaussian count. The cap is optional and off by default, these are our development scenes, and WorldMirror's own camera pass sees the held-out images (Appendix J.6). On point centers with merging off, uniform also leads by +44.27 F-score points at the tight budgets we tested and by +2.29 at the shipped cap (Table 28). This applies to readers like those you describe who prune VGGT or other feed-forward predictions to a tight point budget by confidence ranking. The rule is to filter by confidence, draw the candidate pool uniformly instead of taking the top-K, then apply the same spatial sampler.

[Edited 18 Nov: this paragraph is updated.] Among the feed-forward pipelines we read, WorldMirror's optional global confidence cap, off by default, is the one released option that ranks by confidence to a budget, and it loses when switched on. You ask which deployed pipelines are "doing it wrong". VGGT's COLMAP export and the GLB export of DA3 threshold and then draw uniformly, the allocation our matched test supports. The Table 3 gap in the general comment shows what ranking costs once a builder chooses it. The loss is set when the pool is chosen: when the pool's ceiling is below a same-budget comparator, no sampler that keeps its points can recover it. The coreset work you mention, which we cite, establishes that a pointwise score need not give coverage. Our matched test measures how much that costs for reconstruction confidence, and our ceiling can be computed before any sampler runs (Section 3.2). Our new reply to Tr87 adds a context-only ETH3D rendering test, in which no held-out image enters the network.

Retention is a working rule at native tolerance. At 5% and 10% retention random leads on both DTU and BlendedMVS, with intervals that exclude zero (Tables 10 and 11). The crossover moves with dataset, eligibility and tolerance, so we state the rule by retention rather than by one pool-size threshold.

The DTU comparison holds across the eligibility thresholds and tolerances we tested (Appendix J.1). Across six confidence cuts on 21 DTU scenes, the random pool leads by 48.28 to 54.50 points, and eligibility from the released pointmap percentile mask alone gives a similar lead at native tolerance (Table 22). At half, native and double tolerance, the ceiling of the narrow ranked pool (four candidates per output point) stays below the score of ALL (FPS on the full eligible set) in every DTU condition at the main cut. BlendedMVS depends more on tolerance. At the main cut the ceiling rules out all conditions at native and double tolerance and some at half tolerance, where random's narrow-pool lead stays positive at +10.10 (95% range 3.94 to 19.29). With the mask alone at half tolerance the difference is +1.63, from -10.50 to +10.81, too close to call.

Two points in your summary. On BlendedMVS the intervals at 25% and 50% retention include zero, and ranking leads at 75%, where its interval excludes zero (Table 11). On DTU at the main cut random leads at every retention we tested (Table 10). The roughly 97% figure you cite is our measurement of confidence-proportional drawing on VGGT outputs, the selection step alone, not the full Mini-Splatting or Taming systems (Appendix E). Drawing in proportion to a score is not ranking, so that recovery agrees with our rule. Does that answer whom the rule helps? On DTU at tight budgets, ranking removes scene support that no sampler selecting from the ranked pool can restore, in 84 of 84 VGGT conditions (Table 5). Our matched test and ceiling let a builder with reference geometry check this offline before choosing a sampler.

AuthorsOpenReview ↗

Follow-up (18 Nov): retention forecast result

Thank you again for your review. This follows up our 15 Nov reply with a new result. You asked when random selection should be preferred. Before a new ETH3D retention run, we fixed a forecast from Tables 10 and 11: the random pool would beat the ranked pool at 5% and 10% retention. It did at both. Here retention is the share of eligible points kept in the candidate pool.

ETH3D retentionRandom minus ranked, F-score points95% interval, scenes resampledForecast
5%+16.90[7.55, 27.53]random leads: held
10%+15.03[6.97, 24.37]random leads: held
42%+2.39[-0.96, 5.94]none; too close to call

So at the low retention levels we tested, filter by confidence and then draw the pool uniformly.

Reviewer_Tr87OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether the most reliable 3D prediction is the best reserved point under the fixed reconstruction budget. Using the same downstream sampler and output budget, the authors compared equal size candidate pools formed by confidence ranking or uniform sampling. The results show that confidence ranking can make some reference surfaces have no adjacent candidate points. Using reference geometry, they derived the upper limit of the F-score that any sampler could achieve by selecting points from a given pool without moving the points. In multiple test settings, mixing uniform sampling points into the sorting pool can recover most of the lost coverage.

Critical strengths

1.This paper makes a practical distinction: confidence helps to eliminate unreliable points, but it is not a good criterion for allocating fixed reconstruction budgets.

2.The matching pool comparison method can isolate the influence of sorting, while the coverage boundary indicates when the sampler cannot recover the missing coverage only by the reserved points.

3.The evaluation covered multiple datasets and reconstructed models. The mixing tank experiment also shows a simple remedy: retain some randomly selected candidates.

4.The rebuttal answers my concern about rendering. Under the condition of equal Gauss counting, uniform selection improved the reserved view peak signal-to-noise ratio (PSNR) by 3.51 dB on average in all 10 tested ETH3D scenes.

Critical weaknesses

1.Previous studies have shown that confidence pruning may leave loopholes. The main progress of this paper is to diagnose when the loss is irreversible, rather than introducing a new selection method.

2.The overlay boundary requires reference geometry and is only applicable to samplers that keep the candidate coordinates fixed. It cannot be directly used in new scenes without real values.

3.The preferred policy depends on the number of points reserved. On BlendedMVS, uniform sampling performed better when retaining 5% and 10%, but confidence ranking performed better when retaining 75%. As for the unseen scene, it is not clear how to choose between the two.

Questions for authors

Q1. Practical relevance to 3D Gaussian Splatting. The paper motivates its problem partly through fixed primitive budgets in Gaussian Splatting. Can the authors demonstrate that the observed geometric coverage advantage of random or mixed sampling translates into improved downstream rendering quality under identical Gaussian budgets? In particular, do the conclusions remain valid when evaluating PSNR, SSIM, and LPIPS rather than point-cloud F-score?

Q2. Comparison with spatially aware confidence selection. Why is uniform random sampling the primary alternative to confidence ranking? Have the authors considered confidence-weighted FPS, voxel-based confidence selection, spatially stratified sampling, or confidence-aware Poisson-disk sampling? Could these methods preserve the reliability advantages of confidence while avoiding the coverage loss identified in this paper?

Q3. Role of multi-view redundancy. Since predictions at identical or nearly identical 3D locations are counted as separate candidates, how much of the observed effect is attributable to redundant predictions across different views? Could the authors provide an ablation that removes duplicate or near-duplicate predictions, or performs view-balanced sampling before constructing the candidate pools?

Q4. Generalization across scene types. Can the authors provide additional experiments on larger and more diverse indoor and outdoor datasets? In particular, how does the phenomenon change with scene complexity, spatially nonuniform geometry, occlusion, and varying camera coverage?

Q5. Practical use of the coverage ceiling. Since the coverage ceiling requires reference geometry, how can it be used in an actual inference-time reconstruction system where ground-truth geometry is unavailable? Can the authors propose an observable surrogate that estimates the risk of losing geometric coverage without requiring reference geometry?

Q6. Tightness of the coverage ceiling. The coverage ceiling assumes perfect precision when deriving the upper bound. How tight is this bound in practice, particularly for wider candidate pools where it no longer rules out recovery? Could the authors investigate whether a tighter upper bound incorporating the precision distribution or spatial distribution of the selected candidates can be derived?

Q7. Adaptive selection strategy. The results suggest that random sampling is preferable at narrow candidate-pool widths, whereas confidence ranking can become preferable at wider widths. Can the authors develop an adaptive strategy that determines the appropriate mixture of confidence-ranked and random candidates based on measurable properties of the candidate population, rather than relying on manually selected mixture ratios?

Q8. Generality across reconstruction architectures. The experiments cover several reconstruction backbones, but some models share pretrained components or are evaluated under different eligibility rules. Can the authors provide a more systematic comparison using independently trained architectures, consistent eligibility definitions, and identical evaluation protocols?

3 comments

You ask what "fundamentally new understanding is obtained beyond the established observation that aggressive score-based pruning can produce redundant samples and reduce coverage". Prior work (Section 2) reports holes after confidence downsampling, and the core-set selection you mention shows score-ranking losing to random; what we add is a matched test that measures what confidence ranking costs at a fixed budget, and a per-scene ceiling that rules out recovery by any sampler selecting from the ranked pool, in 84 of 84 VGGT DTU conditions (Table 5).

What we add is where the loss sits: the ranked pool lacks the surface itself. Exact deduplication removes no candidates on our 21 DTU scenes, and FPS already spreads the ranked pool, yet it scores 23.52 F-score against 77.62 for an equal-size random pool, and its ceiling of 24.23 rules out recovery from the ranked pool (Table 3). Because a subset cannot cover more reference surface than its pool, our test rules out every coordinate-preserving sampler that selects from the ranked pool, before any of them runs. The point-selection work we review in Section 2 changes the sampler; we change only the pool. So what is new is where the loss sits and whether it can be recovered: the ranked pool lacks the surface itself, and our ceiling shows that no sampler selecting from that pool can restore it, in 84 of 84 VGGT DTU conditions.

Q1. At identical Gaussian budgets, our random pools render better, end to end in WorldMirror and in both of our own fits. We hold out every eighth view, build splats from the other 43 and render through WorldMirror's own path. At equal Gaussian count, our uniform selection renders held-out views +1.25 dB PSNR better than ranked (95% range 0.87 to 1.61, 18 of 21 DTU scenes), with the cap optional and off by default, merging on, no optimization step, development scenes and held-out images entering its camera pass. Without thinning it leads by +3.27 dB (2.89 to 3.66, 21 scenes). With nothing fitted, the finding is the paired difference (11.73 against 15.00 dB; Table 28, Appendix J.6). In our own fits, VGGT produces geometry and cameras from all views; Gaussian initialization and fitting use only training views (Appendix J.2). Our reduced fit (3,000 steps, fixed poses, no densification, one seed) gains +4.19 dB over 28 scenes at equal budgets, with SSIM and LPIPS moving the same way (Table 23). Our fuller fit (30,000 steps, densification, equal final counts, one seed) gains +1.20 dB (0.67 to 1.72), again with SSIM and LPIPS, and the DTU scenes carry it. BlendedMVS alone gives +0.29 dB, from -0.93 to +1.53, too close to call. There the final Gaussian centers score 24.29 F-score for random against 33.60, after pruning, cloning and splitting rather than from the starting pool.

Q2. On DTU, confidence-weighted FPS on a uniform pool scores within 0.3 points of plain uniform FPS, and the pools that confidence builds score lower. The one confidence-weighted FPS we tested reaches 77.35 mean F-score on a uniform pool, against 77.62 for plain uniform FPS (Table 24). When confidence builds the pool instead, the score falls, to 67.35 for a voxel-confidence pool and 29.80 for a stratified-confidence pool. Confidence-ordered greedy Poisson on a uniform pool reaches 69.87, so how confidence enters the sampler matters too (Appendix J.3). Random is the matched-test control, not a new method, and uniform FPS is already coverage-aware as the greedy k-center rule (Section 3.2).

Q3. We ran both ablations you ask for (Table 25). Joint voxel and view balancing cuts our gap from 56.57 to 9.97 points, against 13.36 for equal-count random thinning alone. Near-duplicate removal changes the gap by +1.17 relative to its thinning control, from -2.81 to +4.74, too close to call. On views, our DA3 pools at four candidates per output point give the ranked pool 14.58 effective views against 46.99 for random (Appendix J.3). MapAnything's ranked pool is about as concentrated, yet its ceiling gap is too close to call (Appendix J.4), so we measure the loss by surface support, through our ceiling.

Q4 and Q8. The random pool leads with DA3, with DUSt3R on supplied cameras, and on the pooled added scenes (Table 26, Appendix J.4). Of 21 further public scenes we tried from 7-Scenes, 12-Scenes and two further ETH3D scenes, 10 passed our fixed entry checks on camera alignment and reference points. On them random leads by +33.54 points (95% range 22.49 to 43.50) as one pooled estimate. DUSt3R is architecturally independent of VGGT and shows the same loss in our diagnostic runs. We keep each model's own eligibility rule, because each backbone's confidence defines the pool its users would rank, and within each rule our matched test changes only pool membership (Appendix C).

Q5. Retention is the observable guide, and none of four other reference-free statistics predicts the gap better. The ceiling needs reference geometry, so it is our offline check (Section 3.2). We tested four reference-free statistics against retention on the seven BlendedMVS scenes, one scene left out at a time, and retention has the lowest error, 170.42 mean squared error against 178.49 (Appendix J.7). At 5% and 10% retention random leads on both DTU and BlendedMVS (Tables 10 and 11).

Q6. The ceiling sits close to what FPS achieves on ranked pools. On DTU our mean observed slack grows from 0.71 to 3.39 points as the pool grows from four to thirty-two candidates per output point (Appendix J.7). On BlendedMVS at thirty-two, the ceiling is 64.93 against an achieved 61.55. On the full eligible set it sits well above FPS, as expected for a pool that keeps all support (Table 9). A tighter ceiling that uses the exact budget and a lower bound on points failing the precision tolerance gives the same value on every DTU scene, because every pool we tested keeps the precision cap at one. A bound using the spatial distribution of the selected points could lower the ranked-pool ceiling by at most that slack, since no valid ceiling falls below what FPS achieves.

Q7. An adaptive mixture agrees with the fixed random rule: +0.007 F-score points, from -0.021 to +0.031 (Table 29). It chooses its ranked fraction on the remaining development scenes, leaving one DTU scene out at a time. Retention (Q5) says when to change it.

Of the 38 interval comparisons in Table 21, 34 exclude zero before any correction. We resample scenes, average FPS starts within each scene, and treat the comparisons as dependent (Appendix I). The four that include zero are BlendedMVS at 25% retention, BlendedMVS mixed minus random, and MapAnything's ceiling gap at both pool sizes. By regime, on DTU at the main cut random leads at every retention we tested, and on both datasets it leads at 5% and 10%. On BlendedMVS the intervals at 25% and 50% include zero, and ranking leads at 75% (Tables 10 and 11).

ETH3D checks the effect outside VGGT's training sources and our protocol development. BlendedMVS is on VGGT's disclosed training list, and we developed the protocol on DTU, leaving out the one scan inspected during development (Appendix H). Because of its camera residuals, we use ETH3D to test the sign of the effect. On the ten scenes with usable predictions, random exceeds ranked at both output budgets and both matched-pool definitions (Section 4.4, Table 6).

AuthorsOpenReview ↗

Follow-up (18 Nov): context-only rendering test (Q1)

Thank you again for your review. This follows up our 15 Nov reply with a new result. You asked whether the coverage loss matters for rendering at the same Gaussian budget. It does: uniform selection renders better at the same Gaussian count when WorldMirror receives only the context images. Its optional confidence cap, off by default, is switched on. There is no per-scene optimization. One similarity transform, fitted on the context camera centres, places the prediction in the benchmark frame, and held-out views are rendered from their benchmark cameras, the same for both arms. We compare uniform and confidence-ranked selection at equal Gaussian count on ten ETH3D scenes, with six fresh seeds per scene.

ETH3D comparison or checkPSNR difference (dB)95% interval, scenes resampledScenes favouring uniform
Uniform minus ranked, equal Gaussian count+3.51[2.38, 4.66]10/10
Seed-split null (should be zero)-0.01[-0.02, 0.01]—
Planted +0.20 dB shift (should be recovered)+0.19[0.18, 0.21]—

A second checker recomputed the result from the saved per-view renders. At 50% point retention the gain is +4.30 dB [3.19, 5.40], again on all ten scenes. Unlike our earlier DTU test in the 15 Nov reply, no held-out image enters the network here. We would welcome your view on whether this answers Q1.

Reviewer_nEc1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the trade-off between pointwise confidence and spatial coverage in 3D reconstruction. Through extensive experiments, the authors demonstrate the limitations of confidence-based ranking and introduce a coverage ceiling to quantify irreversible coverage loss.

Critical strengths
  1. Well-Controlled Experimental Design: The matched-pool evaluation effectively isolates the impact of confidence ranking under fixed sampling conditions.

  2. Useful Diagnostic and Ablation Analysis: The coverage ceiling identifies irreversible support loss, while the ablation studies clarify the effects of sampling budgets and selection strategies.

Critical weaknesses
  1. Limited Novelty and Theoretical Insight: The confidence–coverage trade-off is well-established in spatial sampling. While the experiments validate this phenomenon, the paper provides limited theoretical insight into why or when confidence ranking fails.

  2. Limited Practical Significance: Although the diagnostic analysis is useful, existing methods already employ randomized or diversity-aware sampling. The paper does not demonstrate clear improvements over existing pipelines or in downstream reconstruction quality.

Questions for authors
  1. Conditions for Confidence Ranking Failure: What factors determine when confidence ranking becomes harmful? How do confidence reliability, geometric accuracy, and spatial distribution explain the different behaviors across reconstruction models?

  2. VGGT-Ω Results: Why does VGGT-Ω exhibit the opposite trend, with confidence ranking outperforming random sampling? Could this be related to confidence estimation, reconstruction quality, or evaluation settings?

  3. Downstream Reconstruction Quality: Does improved spatial coverage translate into better end-to-end reconstruction or registration accuracy? Can the authors demonstrate benefits beyond point-cloud F-score, particularly when random sampling reduces precision?

  4. Generalization to Large-Scale Scenes: Would the conclusions hold for large-scale aerial or UAV reconstruction, where non-uniform view coverage and geometric complexity may significantly affect the confidence–coverage trade-off?

  5. Presentation and Readability: The paper contains too many short paragraphs and subsections, which interrupt the reading flow. Consolidating related discussions would improve coherence. Additionally, Figure 2 is overly simplistic and would benefit from a more polished and informative visual design.

2 comments

Under the evaluator we use for all our other DTU results, random leads ranked on VGGT-Omega by +5.76 points (95% range 2.00 to 8.79) and wins 19 of 21 scans (Appendix E, "VGGT-Omega checkpoint", and Table 21). An error in one of our evaluation scripts caused the VGGT-Omega reversal. The faulty script queried nearest neighbours in the opposite direction, so masked predictions and plane-filtered reference points swapped roles in precision and recall, which flipped the sign of our submitted figure. No other result in the paper used that script. The main evaluator reproduces our main DTU means, and an independent rebuild of all ranked and random selections gives the same corrected VGGT-Omega numbers. That score-based selection can lose to random or leave holes is known from earlier work; what we add is a matched test that measures what confidence ranking costs at a fixed budget, and a per-scene ceiling that rules out recovery by any sampler selecting from the ranked pool, in 84 of 84 VGGT DTU conditions (Table 5).

On presentation, Figure 2 in Section 3.1 shows how each pool reaches the surface, and Section 5 closes with an "In practice" paragraph that gives the practical advice.

We measure what ranking costs and when no sampler can repair the loss. In the Table 3 gap in the general comment, the ranked pool's ceiling sits less than a point above its own score, so no sampler that keeps the points can close the gap. You credit the "Well-Controlled Experimental Design". Randomized and diversity-aware samplers act after the pool is fixed, and our ceiling tells a builder when no such sampler can repair it.

The random pool also renders better and has lower registration error on disturbed DTU. In the reduced fit at equal Gaussian budgets, random gains +4.19 dB PSNR, and SSIM and LPIPS also improve (Table 23). After our fuller fit with densification the gain is smaller, +1.20 dB (95% range 0.67 to 1.72), and DTU drives it. Our two fits differ in steps, densification and count matching, and each gain holds within its own fit. We also ran WorldMirror's optional cap (off by default) end to end at shipped settings, merging on, no optimization. Uniform renders held-out views +3.27 dB better (2.89 to 3.66, 21 DTU development scenes), and +1.25 dB (0.87 to 1.61) at equal Gaussian count, though held-out images enter its camera pass (Appendix J.6). On disturbed DTU, rotation error is 1.05 degrees with random against 4.95 with ranked, measured against a camera-derived reference rather than independent ground truth (Appendix J.5). Across all scenes, mean center precision is 75.11% for random against 78.90% for ranked, and mean recall 77.05% against 38.54%. On the seven BlendedMVS scenes its center precision is much lower, 54.95% against 74.74%, yet random still leads there on PSNR in the reduced fit (Appendix J.2).

Ranking hurts most when the pool is narrow and predictions reach the surface. Ranking can remove only surface the predictions reach, and in our results the harm grows as the pool narrows. Where predictions reach the surface, the mean gap between the ceiling and ALL's F-score is large: 35.54 with DA3 and larger with DUSt3R on supplied cameras (Table 21). MapAnything's ALL F-score is 13.97 under our estimated-camera diagnostic, an error level at which it recovers little surface to lose. Against that baseline the gap between ALL and the ceiling is +0.71, from -4.54 to +5.44, too close to call (Table 26, Appendix J.4). On VGGT-Omega, 8.3% of eligible predicted points fall within the 5 mm tolerance and the contrast is smaller. Within each model, our matched test changes only pool membership, so ranking causes the loss wherever the interval excludes zero. Across models the loss tracks reach. It is large where the full eligible set already scores well and near zero for MapAnything. Yet DA3 and MapAnything concentrate their ranked pools about equally (Appendix J.3). A published audit of seven feed-forward backbones, which we cite in Section 2, finds that their confidence ranks pointwise error well. That addresses calibration, not coverage, and a well-ranked score can still remove the only candidates covering part of a scene. For deciding when ranking is unsafe we give two tests. A pool whose ceiling falls below a same-budget comparator is ruled out, and otherwise retention is the guide: random leads at 5% and 10% retention on both DTU and BlendedMVS (Tables 10 and 11).

The random pool leads on further backbones and public scenes. It leads with DA3 and with DUSt3R given supplied cameras (Table 26, Appendix J.4). It leads on DTU, on BlendedMVS and on 10 pooled scenes from 7-Scenes, 12-Scenes and ETH3D. For aerial or UAV capture, our ceiling is the check to run. It needs reference geometry with fixed alignment, masks and tolerance (Appendix F), not a particular capture setup. So a builder with aerial reference geometry can test a pool offline before sampling.

AuthorsOpenReview ↗

Follow-up (18 Nov): independent evaluator check

Thank you again for your review. This follows up our 15 Nov reply with a new result. A separately built evaluator now confirms the corrected VGGT-Omega result. The submitted reversal came from an old scoring script that queried nearest neighbours in the wrong direction, which applied the observation mask and the plane filter to the wrong point sets. So it changed which points were scored, not just which number was called precision. Every Omega selection is now scored with the main evaluator, and no value from the old script remains in the revised PDF. The table compares the separate evaluator with the paper on DTU.

ModelSeparate evaluator F, ranked / randomPaper F, ranked / randomRandom minus ranked, separate evaluator (95% interval)Scans favouring random
VGGT23.52 / 77.6123.52 / 77.62+54.10 [49.79, 57.95]21/21
VGGT-Omega (corrected)15.71 / 21.4715.71 / 21.47+5.76 [2.05, 8.83]19/21

Checks with identical inputs, planted precision and recall values, and swapped selection labels all pass, and a second checker recomputed the means and intervals from the rows. The evaluator's implementer had seen the paper's values, so this is a reproduction rather than a blind test. For large-scale aerial capture, our 15 Nov reply explains how to run the ceiling offline with reference geometry. We are glad to answer any further questions on the evaluation.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Overall response to reviews

[Update 18 Nov: the latest status is in our 18 Nov note "Where each concern stands".]

That score-based selection can lose to random or leave holes is known from earlier work; what we add is a matched test that measures what confidence ranking costs at a fixed budget, and a per-scene ceiling that rules out recovery by any sampler selecting from the ranked pool, in every VGGT condition we tested on DTU (Table 5). Reviewer pxqk says the finding is "directly relevant to anyone pruning VGGT/feed-forward predictions to a point budget". For them we give one design rule. Filter by confidence, then draw the candidate pool uniformly rather than taking the top-K, and only then apply the spatial sampler. On DTU, our ranked pool (top confidence) scores 23.52 F-score against 77.62 for an equal-size random pool (Table 3). No sampler that keeps the ranked pool's points can score above 24.23, its coverage ceiling. Our rule is for the tight-budget settings we tested, for example when a builder turns on an optional ranking cap such as WorldMirror's.

All four reviewers credit our matched pool test, and pxqk calls our ceiling "a genuinely useful contribution". The test changes only which points the sampler may choose from. The ceiling bounds the F-score of any sampler that selects from a pool without moving its points, under fixed alignment, masks and tolerance. It holds because a subset cannot cover more reference surface than its pool. It is an offline check. With reference geometry, if a same-budget comparator exceeds a pool's ceiling, we rule that pool out before any sampler runs. Below, retention is the share of points kept.

Confidence still works as a filter. A 50/50 mixed pool recovers 99% of the ranked-to-random gap at the smaller DTU budget (Section 4.5).

With our main DTU evaluator, random also leads on VGGT-Omega, on 19 of 21 scans. The submitted reversal came from an error in one of our evaluation scripts that flipped its sign; we have fixed it, and our revised PDF reports the changed numbers (Appendix E).

  • Novelty: earlier coverage and hole-patching work ask whether a score-chosen subset covers the data and how to patch holes; our ceiling asks whether any sampler can recover a ranked pool's loss (pxqk, Tr87, nEc1, Section 3.2, Table 3).
  • Real systems: the VGGT and DA3 exporters already filter and then draw uniformly. Where a pipeline offers ranking, as WorldMirror's optional cap does (off by default), our uniform selection leads on point centers at the tight budgets we tested (Table 28). We also ran WorldMirror end to end at the shipped cap, with voxel merging on and no optimization step. There our uniform selection renders held-out views 3.3 dB better on our DTU development scenes, and still 1.2 dB better when thinned to the ranked Gaussian count, though WorldMirror's camera pass sees the held-out images (pxqk, ciBa, Tr87, Appendix J.6).
  • Beyond F-score: in our tests random also renders better (after densification, from the DTU scenes) and has lower registration error on disturbed DTU against a camera-derived reference (Tr87, nEc1, ciBa, Appendix J.2 and J.5).
  • When ranking fails: the harm grows as the pool narrows where predictions reach the surface. We give two tests for when ranking is unsafe, the ceiling and a retention rule (nEc1, ciBa; Sections 3.2, 4.2 and 4.4).

What changed in the PDF: we added these analyses in Appendix J and moved the rendering and pipeline results into the main text.

AuthorsOpenReview ↗

Where each concern stands

This updates our 15 Nov "Overall response to reviews" with the results added since. The revised PDF now includes these results as Appendix K: context-only rendering on ETH3D (K.1), the ETH3D retention forecast (K.2) and the separate DTU evaluator (K.3).

Thank you to all four reviewers. In one line, confidence is not coverage. Keeping only the most confident points to meet a budget can leave whole parts of a scene with no points, and no sampler that picks from those points can bring them back.

Here is where each concern stands.

  • What is new is a per-scene test (Tr87, pxqk, nEc1). Earlier work saw holes after pruning. With reference geometry, our ceiling shows when, for a given scene and budget, no sampler that keeps the ranked pool's points can recover the loss. A forecast we fixed before a new ETH3D retention run also held, as our reply to pxqk shows.
  • The loss shows up in rendering (Tr87, ciBa, pxqk). With no held-out image reaching the network, uniform selection gives higher held-out PSNR on all ten ETH3D scenes at the same Gaussian count. Our reply to Tr87 gives the protocol and checks.
  • The corrected numbers stand (nEc1). The corrected VGGT-Omega result holds, and a separately built evaluator reproduces it, as our reply to nEc1 shows.
  • Tight budgets are where this matters. Feed-forward models predict a point per pixel, and Tr87 notes that "aggressive point or primitive reduction is necessary for computational and memory efficiency". At the tight 5% and 10% budgets we tested, random pools beat ranked pools on both datasets (Tables 10, 11). VGGT's COLMAP export and DA3's GLB export already filter by confidence and then draw uniformly; builders who set a tight budget, for example with WorldMirror's confidence cap, can do the same.

We are glad to answer anything further.

AuthorsOpenReview ↗

TL;DR for anyone building on this work

A short version for readers who want only the takeaway; the details are in "Where each concern stands" and in the reply threads.

TL;DR: confidence is not coverage.

Some AI models turn several photos of a scene into a 3D model made of many points, and they give each point a confidence score. Later steps often keep only a fixed number of these points. The obvious choice is to keep the most confident ones. But confidence judges each point on its own, not how well the points cover the scene together. So keeping only the most confident points can leave whole parts of the scene with no points at all, and no later step that picks from those points can fill the gaps.

A simple fix works better: first drop the points the model is unsure about, then pick from the rest at random, then thin them as usual. In most of the models and datasets we tested, this keeps much more of the scene when only a few points can be kept. That is the situation when these models must run with limited memory or computing time. In a separate rendering test on ten scenes, new views made from it were more accurate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.