acceptodds
Under review as a conference paper at ICLR 2027

View-Structured Conformal Prediction for 3D Gaussian Splatting

Abstract

3D Gaussian Splatting (3DGS) renders novel views in real time, but an uncertainty heatmap only ranks pixels. It does not say how wide an interval must be for a view to meet a coverage target. We treat novel-view synthesis as structured regression and ask that, with probability at least , RGB prediction boxes cover at least a fraction of pixels in a new view. We propose View-Structured Conformal Prediction (VSCP). It splits the pre-calibration scale into a spatial shape from the renderer and a transferable view-difficulty factor, which predicts the smallest view-wise multiplier that shape needs. A held-out quantile over views (View-CP) then gives finite-sample validity even when transferring to new scenes. The same factorization makes the analysis exact. A conformity score is the ratio of oracle to predicted view difficulty, and excess width separates into a test-side and a calibration-side term. Across 13 real scenes, pixel-pooled calibration reaches 89.9% marginal pixel coverage but only 61.4% view-event coverage at a 90% target, while View-CP reaches 91.7–92.0%. At matched coverage VSCP cuts width by 22.1% against a constant scale, and matches a ten-model ensemble's 21.0% reduction using only one model per scene and four rather than ten rasterization passes per query. VSCP also improves on the closest single-model baseline, the 3DGS-U field, by 4.7 points (). The view predictor transfers from bounded source families to all nine unbounded Mip-NeRF360 scenes. There VSCP saves 23.5% width on average and beats the constant scale on all nine scenes. It also keeps a 20.6% saving under a different densification backbone and runs at 216–280 FPS on an RTX 4090.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, 3D Gaussian Splatting, uncertainty quantification, distribution-free inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2w8HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Existing methods for uncertainty estimation for 3DGS produce heatmaps/score that rank pixels from most to least certain based on the area under the sparsification curve (AUSE) and rank correlation. This ordering does not contain any information about how wide the error interval could be. L43-46 gives an example of methods that miss 10 % of the pixels: one method misses 10 % in all views, and another method misses 20 % in half the views and 0 % in the rest, but their pooled coverage is the same.

The paper asks a question for each new view: "With probability at least 1−α1−\alpha, does a new rendering cover at least a 1−β1-\beta fraction of its pixels?" The main contributions are as follows:

  • The authors split uncertainty into two parts: a) spatial shape from the renderer, which denotes which pixels within a view are harder, and b) difficulty factor for each view, which denotes how hard that view is.
  • Reusable view difficulty factor which needs to be trained on source views and then can be reused for any new scene.
Critical strengths
  • [S1] The authors present enough comparisons against pixel-pooled CP in Fig. 2 and Fig. 4. This further motivates the main claim of the paper.
Critical weaknesses
  • [W1] The writing is too dense and convoluted. Further, the Conclusion section is just 4 sentences and restates the general motivation. It lacks the structure required for a manuscript submitted to a top-tier conference.

  • [W2] Saving % is the main result metric discussed by the authors. However, it is not defined explicitly anywhere in the manuscript.

  • [W3] In Table 2, the learned view factor aa beats the per-view average bˉv\bar{b}_v by only one point. The authors did not present results for a variant bˉv×b~\bar{b}_v \times \tilde{b}?

Questions for authors
  • [Q1] Why Vanilla 3DGS only on just 3 scene views?
  • [Q2] Does the approach work for other representations like 3DGS-MCMC?
  • [Q3] How robust is the claim in L429-430? The authors claim that VSCP transfers without any fitting to Mip-NeRF 360 scenes. All the scenes in the Mip-NeRF 360 dataset are around an object. Do these results transfer to scenes in ScanNet++ or DL3DV-10K scenes?
  • [Q4] Does this approach work for sparse-view setting ?
  • [Q5] How sensitive is the approach to changing lighting conditions in the scene?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 3 (2/2)

R3.5 (Q1). Why vanilla 3DGS on only three scenes?

We first used three scenes because vanilla densification produces 5.6–7.7× more Gaussians, and we have now extended the study to all three protocols. Doing so exposed a bug. The prior in Table 11 did not exclude the target scene, so the three original scenes now save 17.7% instead of 20.6% (Correction 1). The view regressor is never refit on vanilla models, and the strict row uses the four-source shrinkage for both backbones.

ProtocolScenesVanilla (%)FastGS (%)Vanilla − FastGS [95% CI], winsAbove constant
Source pool, leave one out (Table 11)617.120.8−3.77 [−8.03, 0.96], 2/66/6
Same family, unseen (Table 14)525.023.3+1.61 [−0.22, 3.57], 3/55/5
Strict cross-family (Table 15)922.5023.66−1.16 [−4.15, 1.53], 4/99/9

Every interval includes zero, so we detect no difference between backbones, but the intervals are too wide to claim equivalence within 2 points. Room drops most (−11.5 points leave one out, −10.3 strict), so a backbone change keeps most of the saving, with room as the exception. View-event coverage is 89.0–93.5%. The lowest value (treehill, 89.0%) is Monte Carlo error of the 200 splits, since the exact mean over all 31,824 splits is 11/12 = 91.7%.

Revision. Table 11 has six scenes (L1297), and Table R3 (Appendix D.7, L1310) adds the two transfer protocols. Appendix D.7 gives the treehill check (L1291–1295), and Appendix D.9 the corrected room value (L1345).

R3.6 (Q2). Other representations such as 3DGS-MCMC

We have not tested 3DGS-MCMC, so we make no claim for it. The method needs only a trained 3DGS whose Gaussians can carry the residual field and the support statistics. 3DGS-MCMC changes how Gaussians are created and moved, the same kind of change as our vanilla study, which does not cover it. The guarantee holds for any fixed scale (Theorem 1), but we have no width results for it.

Revision. Appendix E lists 3DGS-MCMC and other density-control schemes as untested (L1619–1620).

R3.7 (Q3). Scenes that are not object-centric (G3)

On three DL3DV-10K scenes, validity and the saving over the constant both transfer without refitting, but the advantage over field-based baselines does not. ScanNet++ needs a data license that we could not obtain in time. Before training we fixed the selection rule. In each of the indoor, urban and natural categories, we took the first scene in hash order whose camera path spans less than 180° in azimuth. Each scene uses one FastGS model at 960×540, the strict prior and four-source shrinkage.

Saving (%)Coffee-Shop (78°)Square (90°)Botanical (96°)Mean
Ours a × b̃24.3330.9412.3022.52
3DGS-U18.9624.6717.1520.26
b̄_v × b̃, source-selected (1.0, 0.25)18.4127.317.1917.63
b̄_v × b̃, (λ_a, λ_b) = (0.75, 0.5)20.6127.9416.4621.67

Ours beats the constant on 3/3 scenes and 3DGS-U on 2/3 (+2.26 [−4.85, 6.27], not significant), against +5.06 on Mip-NeRF 360. Against b̄_v × b̃ with λ_b = 0.5 the margin is only +0.86. On Botanical-Gardens the transferred view factor hurts (alone it is 8.77% wider than the constant). A post-hoc diagnostic (Appendix D.16) explains this scene and kitchen, but it needs test labels. View-event coverage is 91.9–93.8% for every row. Averaged over all splits it equals 13/14 for any scale, so it checks the code rather than the method.

Revision. New Appendix D.15 (L1527, Table R6) and D.16 (L1554). Section 6.4 states the result and its limits (L476–477).

R3.8 (Q4). Sparse-view setting (G3)

View-CP is not designed for the sparse-view setting, because it needs at least m = 9 labeled held-out views at α = 0.1, the floor of any finite split-conformal threshold (Appendix C.5). With very few captured views, holding out nine would remove most of the training data. When extra labeled views exist, the guarantee still holds for a sparse-view model, but we have not measured the width there.

Revision. Appendix E states this requirement and names the sparse-view setting as out of scope (L1613–1616).

R3.9 (Q5). Changing lighting

The answer depends on whether the lighting change breaks exchangeability, and we have not run a lighting experiment. If calibration and test views come from the same capture, the guarantee holds by Theorem 1, and changing light affects only the width. If lighting differs between calibration and test views, as in photo collections taken over time, exchangeability fails, and the arc result of R1.1 shows that coverage can then drop well below target. Such data also needs an appearance model in the center, which is outside our scope.

Revision. Appendix E adds lighting change across captures to the stated scope limits (L1624–1626).

Thanks to the authors for their response. My unaddressed concerns are as follows:

  • [Q2] 3DGS-MCMS uses a more robust densification strategy than the heuristic-based adaptive density control methods described in vanilla 3DGS and Fast-GS. It will be interesting to verify the sensitivity of VSCP across different densification schemes and Gaussian budgets. A similar analysis can be performed for different pruning strategies, such as PUP-3DGS.
  • [W3] The new comparison shows that the learned factor beats  bˉv×b~\bar{b}_v \times \tilde{b} only by a margin. Are there settings in which the learned factor will outperform other variants by a large margin? As you have already concluded, most of the gains come from estimating and shrinking the two parts separately. Further analysis of the learned factor will improve clarity on this mechanism.
Reviewer_ceNoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes View-Structured Conformal Prediction (VSCP) for uncertainty quantification in 3D Gaussian Splatting. The method combines a renderer-derived spatial uncertainty field with a learned view-difficulty factor, followed by view-level conformal calibration. It aims to provide finite-sample coverage guarantees while reducing prediction interval widths.

Although the paper presents a technically reasonable formulation and several encouraging results, I am not fully convinced by its research motivation, methodological novelty, or empirical significance.

Critical strengths

The paper addresses a relevant aspect of uncertainty quantification by distinguishing pixel-level uncertainty ranking from statistically guaranteed view-level coverage.

The proposed factorization of spatial uncertainty and view difficulty is intuitive, and the associated theoretical analysis provides some useful interpretations.

The authors evaluate their method across 13 real-world scenes and provide several baseline comparisons, ablation studies, and cross-family transfer experiments.

Critical weaknesses
  1. Unclear research motivation and insufficient demonstration of significance

My primary concern is that the motivation and practical significance of the proposed problem are not sufficiently convincing.

The paper argues that existing uncertainty maps cannot provide statistically guaranteed view-level coverage. While this limitation is mathematically valid, the authors do not adequately explain why the proposed coverage objective is particularly important for practical 3DGS applications.

Specifically, it remains unclear what practical problem is solved by guaranteeing that a specified fraction of pixels falls within prediction intervals with a prescribed probability. The paper does not demonstrate how such intervals improve downstream applications, such as reconstruction quality assessment, active view selection, or robotic perception.

I acknowledge that a methodological paper does not necessarily require downstream application experiments. Nevertheless, the authors should provide a more compelling justification for why this particular uncertainty objective is meaningful beyond its statistical formulation.

As presented, the paper seems more focused on constructing and satisfying a mathematically well-defined objective than on establishing why this objective represents an important unresolved research problem.

  1. Limited methodological novelty

The proposed framework primarily combines several existing ideas: renderer-derived residual uncertainty, view-dependent scale prediction, and standard split conformal calibration.

The finite-sample coverage guarantee largely follows established conformal prediction results once an appropriate view-level conformity score is defined. The excess-width decomposition is mathematically useful, but it appears to follow relatively directly from the multiplicative scale formulation and properties of order statistics.

More importantly, the experiments do not sufficiently isolate the contribution of the learned view-difficulty predictor.

In Table 2, the learned view factor alone achieves a 13.6% width saving, compared with 12.6% from the original residual field's view-wise magnitude. This raises the question of whether the full method's improvement primarily comes from the learned predictor or simply from independently normalizing and shrinking different components of an existing uncertainty field.

A stronger comparison against a non-learned factorization of the 3DGS-U field would be necessary to establish the incremental contribution of the proposed learning component.

Overall, I find the methodological contribution somewhat incremental relative to the existing uncertainty estimation and conformal prediction literature.

  1. Experimental improvements are not sufficiently compelling

Although VSCP achieves a 22.1% reduction in interval width relative to the constant-scale baseline, the improvement over the stronger existing methods is considerably smaller.

Compared with 3DGS-U, the average improvement is 4.7 percentage points. Compared with the ten-model ensemble, the improvement is only 1.2 percentage points and is not statistically significant.

The reduced target-scene inference cost relative to the ensemble is a meaningful advantage. However, this comparison does not fully account for the additional source-scene meta-training and out-of-fold model training required by VSCP.

Furthermore, the strict cross-family experiment does not include a direct comparison with 3DGS-U, arguably the most relevant single-model baseline. This omission limits the strength of the claimed cross-scene generalization advantage.

The reported results demonstrate that the method can work, but I do not find the improvements sufficiently substantial or distinctive to establish a strong empirical contribution.

  1. Presentation and experimental evidence could be better connected

The paper contains substantial theoretical and experimental material, but I found the overall presentation less convincing than the amount of material might suggest.

Some figures provide intuitive illustrations of the proposed formulation, yet they do not offer much additional evidence about the practical significance of the method. For instance, Figure 3 illustrates the effects of view-dependent scaling, while Figure 5 visualizes calibrated prediction intervals, but neither establishes whether these improvements translate into meaningful benefits for downstream use.

More generally, the paper moves from theoretical coverage guarantees to claims about reliable rendering without sufficiently demonstrating the practical implications of this connection.

I would encourage the authors to focus less on presenting multiple variations of the same coverage and width metrics and more on establishing a clear connection between the proposed objective, its algorithmic contribution, and its practical significance.

Questions for authors
  1. What specific practical applications motivate the proposed view-level coverage objective? Why should this objective be preferred over existing uncertainty or risk measures for 3DGS?

  2. How much of the improvement comes from the learned view-difficulty predictor rather than simply factorizing and independently shrinking the original 3DGS-U uncertainty field?

  3. Why is the strict cross-family evaluation missing a direct comparison with 3DGS-U? Would the proposed method still provide a meaningful advantage under this more rigorous protocol?

Reviewer_gU5WOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

VSCP reframes 3DGS uncertainty as view-level coverage: View-CP calibrates each held-out view's (1−β) order statistic of residual/scale ratios on top of a factorized scale — a spatial shape b̃ built with the 3DGS-U residual-attribution construction, times a per-view factor a learned once on source scenes from 24 label-free features. Validated on 13 scenes (Tanks & Temples, Deep Blending, Mip-NeRF 360; FastGS backbone; every-8th-frame holdout): pooled-pixel CP hits 89.9% pixel coverage but 61.4% view-event coverage; View-CP restores 91.7–92.0%. At matched coverage: 22.1% saving versus the per-scene conformal constant; ensemble 21.0% with claimed parity (gap 1.2 [−4.0, 6.6], p = 1.0) at one model and four packed passes versus ten and ten (measured 2.32–2.56× speedup); +4.7 over 3DGS-U (p = 0.0225); a strict four-scene bounded-family holdout transfers to all nine Mip-NeRF 360 scenes at 23.5% (p = 0.0039, the n = 9 floor, disclosed); 20.6% survives a vanilla-densification swap. Disclosures include the 8-of-13 source-pool overlap and a full baseline-normalization sensitivity grid.

Critical strengths

The right statistical object, plus a demonstrated blind spot in the field's evaluation practice. The central empirical contrast verifies exactly (89.9% marginal pixel coverage versus ~61% view-event coverage across three maps; View-CP within 0.6 points of the exact 91.9% everywhere). Lemma 3 proves per-view AUSE and Spearman cannot see the view factor, and Table 9 makes it concrete: b̃-only and a×b̃ have identical AUSE (0.2479) and Spearman (0.4604) yet save 9.9% versus 22.1% — the factor invisible to the subfield's standard metrics carries more than half the saving. The lemma is elementary (per-view only; cross-view ranking does see the factor, and the paper itself uses cross-view Spearman), but as an indictment of AUSE-centric evaluation it is checkable and durable, and View-CP is a drop-in layer any uncertainty map can use (coverage 91.4–91.7% across all five Table 1 scales).

Evaluation hygiene well above the area norm. Center, residuals, calibration views, and the conformal layer are fixed; only the scale map varies. Baselines get their stronger normalization with the full grid printed (Table 6); the ensemble column shows its own worst case (drjohnson −19.5) beside its best (bonsai +24.7); the cost claim is measured rather than asserted (channel-packing error 6.2e−8). Table 1 recomputes from the per-scene Table 13, all eight sign tests are exact, the FPS arithmetic is exact, and the m-sweep non-monotonicity is correctly predicted by the kα = m regime change. The residual uncertainty is about scope, not veracity.

Critical weaknesses

The protocol only tests interpolation, where the certificate matters least. Every result uses the interleaved holdout; the paper's own GAVIS diagnostic shows test directions are as visible as training directions, so GAVIS's 4.2% row is partly a protocol artifact, and the "angularly aware" contribution is unsupported exactly where it would bite — Table 4 shows the two field log-means dominate the descriptor (0.616 of 0.705 Spearman; removing them drops it to 0.457). The deployable promise reduces to: given 11–13 labeled views withheld from training (a real cost, named in the limitations) and batch-known query covariates, certify views like the capture. No experiment quantifies off-trajectory degradation, and the strict holdout is a low-discrimination transfer test: a same-size in-family source pool scores only 0.6 points higher, and family is confounded with bounded-versus-unbounded capture (also disclosed).

The attribution of the headline gain to the named contribution is incomplete, and one cited comparator is unexplained. Table 2's ladder (9.9 → 12.6 → 13.6 → 17.4 → 22.1) never runs the decisive cell: a_v = b̄_v — the field's own label-free per-view mean — with separate shrinkage. The learned regressor's solo margin over that free statistic is 1.0 point (13.6 versus 12.6; Spearman 0.719 versus 0.677), so the 17.4 → 22.1 step conflates separate shrinkage with the learned factor. The +4.7 over 3DGS-U is also construction-versus-construction inside the authors' own pipeline (the 3DGS-U row is their re-implementation of that construction), fairly tuned but not the official system — a caveat the limitations state for GAVIS and the backbone but not for 3DGS-U. Separately, Han & Dumery (2025), which by the paper's own description "store[s] view-dependent uncertainty as a spherical-harmonic feature of each Gaussian" — the directional half of the borrowed field — appears in no table and is not covered by the stated center-changing exclusion, despite Appendix D.3 proving the harness can ingest an external field.

Questions for authors
  1. Report the missing row of Tables 2 and 15: the full factorization with a_v = b̄_v (separate shrinkage, no regressor, no source scenes), on the 13-scene pool and under strict holdout. If it lands within ~1 point of 22.1%/23.5%, contribution (3) should be reframed and the transfer apparatus loses its point; a clear gap would earn the regressor its complexity. Either answer moves my assessment.
  2. Provide any non-interleaved measurement: hold out a contiguous arc or the most angularly extreme views on 2–3 scenes, calibrate on the rest, and report view-event coverage and width. This is also the only regime where the angular features could show their advertised value.
  3. Either add a Han & Dumery (2025) scale-map row under your shared-rasterizer protocol (as done for GAVIS), or state the exclusion reason; and extend the limitations caveat to say the 3DGS-U row is your re-implementation of that construction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.