acceptodds
Under review as a conference paper at ICLR 2027

Hallucinate, Filter, and Refine: Occluded Surface Reconstruction with Generative Priors

Abstract

Reconstructing reliable 3D surfaces from sparse input views remains challenging when large scene regions are unobserved due to occlusions. Generative models can expand view coverage by synthesizing missing observations, but generated views often contain hallucinated structures, pose errors, and appearance drift that can introduce unreliable supervision. We propose Hallucinate, Filter, and Refine, a reliability aware framework that exploits generative priors while controlling their uncertainty throughout reconstruction. We first establish a geometric anchor from sparse inputs and synthesize multiple candidate view sequences along augmented camera trajectories. We then progressively filter generated observations at both sequence and view levels using patch-wise visual compatibility with input views and consistency with observed geometry. Our Auxiliary Control Module (ACM) estimates patch wise reliability based on feature consistency and reconstruction difficulty. ACM regulates photometric supervision by suppressing uncertain generated regions while preserving reliable information. Since uncertain generated regions can be difficult to identify before sufficient geometric convergence, ACM adaptively activates reliability weighting based on reconstruction progress. This design preserves valid information during early optimization while reducing the influence of uncertain generated regions as reconstruction stabilizes. Experiments demonstrate improved occluded surface recovery while maintaining competitive rendering quality under sparse observations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Surface reconstruction, occlusion, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_GFVwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a sparse-view reconstruction framework. The method first generates multiple novel-view sequences using pre-trained SEVA, and then conduct initial selection on generated views through DINO feature. During scene level optimization, the views are further progressively filtered based on designed strategy. After optimization pipeline, the framework is expected to have better reconstruction quality with sparse inputs such as 2 images.

Critical strengths

For sparse-view reconstruction, the best-of-N selection of diffusion generated novel views is reasonable. While most sparse-view reconstruction methods now require diffusion prior, making this qualitative selection automatic is timely.

Critical weaknesses

However, this paper does not seem to actually solve this problem. The biggest concern is, while the baseline for comparing should be 2DGS + SEVA, in Figure. 3 the proposed method shows no qualitative improvements (or even worse) comparing with 2DGS + SEVA. This leads to another concern that the quantitative results in Table. 1 does not correspond with qualitative results. I wonder why OracleGS has the highest SSIM and S2C-3D has the highest PSNR appear to be quite bad in Figure. 3. I can only imagine the reasons be: 1. Qualitative results are extremely selected cases. 2. The quantitative results are not correctly evaluated. 3. The metrics are not representative under such scenario. Either reason makes the experimental results not persuasive.

Questions for authors

While there might be certain misunderstanding, it would be good if the authors could address my concerns stated in the weaknesses section.

4 comments

We thank the reviewer for pointing out the gap between the rendering metrics and the visual results. Investigating it, we found an error in how our rendering evaluation is described in the paper. We correct it below, report new object-masked rendering metrics, and then address the comparison with 2DGS+SEVA, which concerns geometry rather than appearance.

Correction of the rendering evaluation

Section 4 of the main manuscript and Appendix A.6 of the technical supplements state that PSNR, SSIM and LPIPS are computed with object masks on the sixteen PixelNeRF test views. This description is wrong, and we apologize for the confusion. The values reported in Table 1 of the main manuscript were computed on full images without object masks, over 25 DTU views, one of which is an input view. We recomputed all rendering metrics with object masks and report them below. We will replace the rendering columns of Table 1 of the main manuscript with these values and correct the text.

The new evaluation uses the same renders and the same DTU object masks for every method, over the twenty-four held-out views. PSNR is computed over object pixels, and SSIM and LPIPS are computed with the background set to zero.

Table C1. Rendering metrics on the five scenes of Table 1 of the main manuscript. The first three columns reproduce the values reported in Table 1. The last three columns are the corrected object-masked metrics.

methodPSNR as reportedSSIM as reportedLPIPS as reportedPSNR objectSSIM objectLPIPS object
Ours13.040.3770.59512.000.5800.369
OracleGS13.240.3900.62812.020.5080.453
2DGS+SEVA12.500.3770.60711.890.5280.431
S2C-3D13.820.3660.59812.390.5630.360
GeoQuery13.530.3460.57812.510.5640.350
Difix3D12.630.3780.61311.840.5200.428
2DGS11.550.3290.66411.100.5750.397

Table C2. Corrected object-masked rendering metrics over all eleven scenes.

methodPSNR objectSSIM objectLPIPS object
Ours11.870.5860.356
OracleGS11.220.5070.433
2DGS+SEVA11.500.5330.418
S2C-3D12.270.5660.337
GeoQuery12.360.5630.331
Difix3D11.810.5260.406
2DGS10.780.5820.391

Tables C1 and C2 show that, with object masks, our method achieves the strongest SSIM on both scene sets, while GeoQuery and S2C-3D achieve a higher PSNR and a better LPIPS. The corrected evaluation therefore indicates competitive rendering quality on the object rather than the best rendering on every measure. OracleGS has a high SSIM as reported in Table 1 of the main manuscript but a weaker object-masked SSIM and LPIPS, which is consistent with the impression of the reviewer that some methods with high reported metrics look worse. We do not claim that the corrected metrics explain every visual difference in Fig. 3 of the main manuscript.

Selection of the qualitative view

Fig. 3 of the main manuscript shows the same DTU view, view 25, for every scene and every method. Table C3 gives the object-masked PSNR at this view.

Table C3. Object-masked PSNR at the view shown in Fig. 3 of the main manuscript, for the five scenes shown in the figure.

methodscene 21scene 24scene 110scene 114scene 118
Ours8.4011.7914.3212.8014.99
OracleGS8.4310.6114.389.1115.94
2DGS+SEVA9.3111.2013.1210.1415.60
S2C-3D8.9510.9413.6517.6417.77
GeoQuery9.6610.4813.4716.8817.90
Difix3D7.8212.0412.7210.1414.70

Table C3 shows that our method does not achieve the highest PSNR at this view on any of the five scenes, and compared with 2DGS+SEVA it is higher on some scenes and lower on others. The displayed view is therefore not one where our method renders best, although this does not by itself rule out every form of selection bias.

Comparison with 2DGS+SEVA

We agree that the RGB renderings of our method and 2DGS+SEVA look similar, and Fig. 3 of the main manuscript shows only RGB. Rendering metrics and surface metrics measure different properties. PSNR, SSIM and LPIPS compare images, while Chamfer distance and surface F1 compare the reconstructed surface with the ground-truth scan, which is the main objective of our paper.

Table C4. Geometry over all eleven scenes.

methodAll F1Obs F1Unobs F1All Chamfer
Ours0.8180.8070.81922.12
2DGS+SEVA0.7980.7730.81723.10
OracleGS0.7760.7430.82924.09
S2C-3D0.7190.6670.78924.76
Difix3D0.6490.5950.70930.07
2DGS0.6200.5980.65343.05
GeoQuery0.4930.4450.52642.77

Table C5. All F1 per scene for Ours and 2DGS+SEVA.

scene21243437384082106110114118
Ours0.9500.8460.8020.5340.8410.7370.6030.8320.9310.9670.959
2DGS+SEVA0.9310.7240.8850.5170.8410.7910.5440.9000.8210.8680.951

Tables C4 and C5 show that our method improves on 2DGS+SEVA in overall F1, Observed F1 and Chamfer, while the two perform comparably on unobserved surfaces. The improvement is modest, mainly on observed surfaces, and varies between scenes.

The 2DGS+SEVA baseline uses the same selected SEVA sequence as our method, with all 16 generated views and its own reconstruction settings.

We will state in the paper that Fig. 3 of the main manuscript compares appearance, and that the geometric comparison is given by the surface metrics.

As the discussion period is coming to an end, we would be sincerely grateful if the reviewers could kindly take a moment to consider our rebuttal and the additional experiments. We welcome further discussion and hope the new evidence can be considered in the final evaluation.

Reviewer_dP3gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework for surface reconstruction from two sparse input views. The method firstly uses a feed-forward reconstruction model to obtain the initial geometry and camera poses, and then uses a video diffusion model to generate multiple candidate view sequences. The generated views are filtered at both sequence and view levels. During the final 2DGS optimization, the proposed Auxiliary Control Module estimates the patch-wise reliability of generated views and adaptively controls their supervision. The paper is easy to follow and the geometric reconstruction results are strong.

Critical strengths

This paper targets at an important problem of reconstructing the occluded surfaces from extremely sparse input views. Instead of directly using all generated views as supervision, the method progressively filters unreliable views and controls the contribution of uncertain regions during optimization. The evaluation that separately measures the observed and unobserved surfaces is useful, and the geometric reconstruction results are strong.

Critical weaknesses
  1. I think the novelty of the method is limited. The main pipeline consists of selecting generated sequences with DINO features, pruning views according to their consistency with the current reconstruction, and reweighting the photometric loss based on the difference between the rendered and generated images. These designs are reasonable, but they are relatively straightforward filtering and weighting strategies. OracleGS has already considered confidence estimation for incorporating generative priors into sparse-view reconstruction. Therefore, I think the contribution is mainly a combination of several reliability strategies rather than a substantially new reconstruction method.

  2. I think the experimental evidence is insufficient. The quantitative evaluation is only conducted on five selected DTU scans, while the other datasets only have qualitative results. Moreover, the main components are not separately ablated. The two filtering stages are grouped together as “Filter”, while the reliability estimation and adaptive activation controller are grouped together as “ACM”. Therefore, it is difficult to understand the contribution of each proposed component.

Questions for authors
  1. Why are only five DTU scans selected for quantitative evaluation? Could the authors provide results on all scans or explain the selection criteria?

  2. Could the authors separately ablate the set-level selection, view-level pruning, reliability weighting, and adaptive activation controller?

5 comments

We thank the reviewer for the positive assessment of the problem setting, the presentation and the geometric results, and for the clear requests.

Novelty and distinction from prior work

Generative view augmentation and confidence-aware reconstruction have both been studied before, and OracleGS is the closest prior work. Our framework handles the unreliability of generated observations at several levels, both before and during optimization, rather than only through a confidence weight in the reconstruction loss.

  • Sequence level. Set selection picks one generated sequence from thirty-two candidates by its visual compatibility with the real input views.
  • View level. View pruning removes generated views that disagree with the input views before they are used for training.
  • Patch level. ACM modulates the supervision from the remaining generated views during reconstruction.
  • Optimization level. The adaptive controller decides when this weighting is applied.

The framework separates two decisions. Set selection and view pruning decide which generated observations enter the reconstruction, before optimization. The weighting and its activation decide how strongly the retained observations contribute, during optimization. This design and the effectiveness of each stage are separate claims, and Tables B3 and B4 below measure the latter. Removing generated views has by far the largest effect, although our pruning criterion performs similarly to random pruning. Set selection is better than the mean of a random candidate on the two scenes tested, with a scene-dependent benefit. Reliability weighting and its adaptive activation add small effects within the variation across training seeds. We do not claim that each stage is a new technique on its own, nor that this four-level design is optimal. Our contribution is a framework that makes these decisions explicit, together with an evaluation of observed and unobserved surfaces that measures which of them matter in practice.

Our OracleGS baseline is run from its official implementation in a common setting. Over all eleven scenes our method achieves a higher overall F1 and Observed F1 and a lower Chamfer, while OracleGS achieves a slightly higher Unobserved F1. Because the two pipelines also differ in reconstruction settings, we attribute this difference to the full pipeline rather than to filtering alone. The supported claim is better overall geometry and better preservation of observed surfaces, with unobserved-region accuracy comparable to OracleGS.

Results on all DTU scenes

We agree with the reviewer and now report all eleven scenes of the S-VolSDF DTU evaluation protocol. We will update all results in the final version accordingly.

Table B1. Mean over all eleven scenes.

methodAll F1Obs F1Unobs F1All Chamfer
Ours0.8180.8070.81922.12
2DGS+SEVA0.7980.7730.81723.10
OracleGS0.7760.7430.82924.09
S2C-3D0.7190.6670.78924.76
Difix3D0.6490.5950.70930.07
2DGS0.6200.5980.65343.05
GeoQuery0.4930.4450.52642.77

Table B2. All F1 per scene.

sceneOursOracleGS2DGS+SEVAS2C-3DDifix3D2DGSGeoQuery
210.9500.9150.9310.7720.6620.3270.674
240.8460.8550.7240.7740.8250.7690.646
340.8020.8260.8850.8890.8610.8230.149
370.5340.4580.5170.8050.5690.5970.822
380.8410.8290.8410.8830.8350.8040.415
400.7370.6630.7910.7420.4000.2970.753
820.6030.5340.5440.7580.7170.3060.491
1060.8320.7530.9000.7110.7140.7770.472
1100.9310.8640.8210.4040.4700.6640.210
1140.9670.8850.8680.3280.7390.5860.146
1180.9590.9570.9510.8420.3470.8710.641
mean of 110.8180.7760.7980.7190.6490.6200.493

Over all eleven scenes our method has the best mean All F1, Observed F1 and Chamfer, and OracleGS has a slightly higher mean Unobserved F1. The ranking varies between scenes, and we will discuss the scenes where other methods are ahead in the paper.

Separate ablation of each component

Table B3. Component ablation with our recipe on the same selected SEVA set. Values are the mean and standard deviation over three training seeds, averaged over all eleven scenes. The no pruning and random pruning rows use one training seed, and random pruning averages five random draws of the kept views. The last row is the per-scene standard deviation across training seeds, averaged over scenes and variants.

variantview pruningreliability weightingwhen weighting is appliedAll F1Unobs F1All Chamfer
no pruningnone, all 16 generated views keptnonenever0.7230.71432.38
random pruningrandom 14 of 16 viewsnonenever0.8130.81022.69
pruning onlyours, 14 of 16 viewsnonenever0.818 ± 0.0100.823 ± 0.01223.35 ± 2.32
always onours, 14 of 16 viewsfull, difficulty and consistencyfrom the first iteration0.820 ± 0.0020.819 ± 0.00222.34 ± 0.56
fixed scheduleours, 14 of 16 viewsfull, difficulty and consistencyfrom the end of warm-up onward0.821 ± 0.0050.818 ± 0.00222.30 ± 0.60
full methodours, 14 of 16 viewsfull, difficulty and consistencyadaptive controller0.822 ± 0.0100.823 ± 0.00822.08 ± 0.45
difficulty onlyours, 14 of 16 viewsdifficulty cue onlyadaptive controller0.817 ± 0.0080.818 ± 0.00722.26 ± 0.91
consistency onlyours, 14 of 16 viewsconsistency cue onlyadaptive controller0.816 ± 0.0050.818 ± 0.00222.46 ± 0.29
standard deviation across training seeds, per scene0.0190.0201.54

Table B3 analyzes the contribution of each component under the same generated views and the same recipe. Removing generated views has the largest effect, substantially improving both F1 and Chamfer over the no pruning variant. Random pruning of the same number of views performs at a similar level to our pruning criterion, also when compared at the same training seed, which indicates that the benefit comes mainly from limiting the generated supervision rather than from the specific views that are removed. Adding reliability weighting yields a slightly lower Chamfer, and the adaptive controller, the fixed schedule and always-on weighting perform comparably, as do the single-cue variants. The differences among the weighting and activation variants lie within the variation across training seeds.

For set selection, we trained our method with view pruning and without the weighting module on each of the thirty-two SEVA candidate sets of scenes 21 and 24, and scored all of them with our protocol. A uniformly random pick from the pool corresponds to the mean over these sets.

Table B4. Selected SEVA set against the pool of thirty-two candidate sets, at 15 mm. One training seed per set.

scenemeasureselected setrandom set, mean and stdrank of selected setbest of 32worst of 32
21All F10.9510.941 ± 0.02014 of 320.9610.870
21Obs F10.9400.929 ± 0.01810 of 320.9510.882
21Unobs F10.9610.951 ± 0.03515 of 320.9780.811
21Chamfer9.73710.365 ± 1.57014 of 328.47115.283
24All F10.8570.748 ± 0.0621 of 320.8570.572
24Obs F10.8390.713 ± 0.0701 of 320.8390.525
24Unobs F10.8950.831 ± 0.0451 of 320.8950.707
24Chamfer18.04326.703 ± 4.3531 of 3218.04340.881

Table B4 shows that the benefit of selection depends on the candidate pool. On scene 24, where candidate quality varies strongly, the selected set is the best of the pool and clearly outperforms a random pick. On scene 21, where the candidates differ much less, the selected set lies in the middle of the pool while remaining above the pool mean on all measures. The selection score is only weakly correlated with accuracy on both scenes.

Comments on the paper

11 comments

We thank the reviewer for the careful reading and the concrete requests. Below we answer each weakness and question in turn. For every point we state what the new evidence supports and where it does not support our original claims, and we will revise the paper accordingly. The controller traces requested in Q1 are summarized for all runs in Table A1, and Table A2 shows two representative traces.

Does the implementation follow Eqs. 11 to 14 of the technical supplements

Yes. The implementation monitors the per-image normalized residual of Eqs. 11 to 14 of the technical supplements, with the parameters listed in Table 7 of the technical supplements. We agree with the reviewer that min-max normalization removes the absolute magnitude of the residual, so the signal can stay flat while the absolute error decreases. To check this we logged the controller signal, the unnormalized photometric residual and the activation state every 100 iterations with three training seeds on all eleven scenes.

Table A1. First activation iteration of the adaptive controller, share of post-warm-up iterations with weighting active, and rank correlation between the controller signal and the unnormalized residual, per training seed. All eleven scenes.

scenefirst activation, seed 0 / 1 / 2active share, seed 0 / 1 / 2rank correlation with unnormalized residual, seed 0 / 1 / 2
214100 / 4300 / 93000.73 / 0.51 / 0.110.88 / 0.65 / 0.00
243700 / 6300 / 63000.28 / 0.54 / 0.54−0.15 / 0.67 / 0.60
346500 / 6300 / 65000.14 / 0.25 / 0.230.19 / 0.14 / 0.15
373900 / 4500 / 37000.42 / 0.25 / 0.680.57 / 0.54 / 0.72
389100 / 3700 / 37000.14 / 0.62 / 0.70−0.31 / −0.10 / 0.41
404100 / 3700 / 39000.59 / 0.82 / 0.820.86 / 0.81 / 0.73
823700 / 4500 / 37000.37 / 0.48 / 0.42−0.15 / 0.32 / −0.11
1066300 / 3700 / 63000.17 / 0.79 / 0.540.03 / 0.78 / 0.35
1106500 / 6500 / 67000.51 / 0.51 / 0.480.67 / 0.61 / 0.66
1146100 / 5900 / 39000.56 / 0.59 / 0.790.33 / 0.18 / 0.35
1186900 / 6300 / 63000.34 / 0.48 / 0.280.00 / −0.16 / −0.12

Table A1 shows that the controller signal follows the unnormalized residual closely in some runs and only weakly or negatively in others, and that the first activation varies with the training seed. This partly confirms the concern of the reviewer.

Table A2 shows the traces of training seed 0 for scene 40, where the correlation is high, and scene 38, where it is negative. The signal is the input of the controller in Eqs. 13 and 14 of the technical supplements, and the residual is the unweighted photometric error over all training views. Active is the share of each window in which the weighting was applied, and the opacity reset occurs at iteration 5500.

Table A2. Controller traces for training seed 0, every 500 iterations after warm-up.

iterationscene 40 signalscene 40 residualscene 40 activescene 38 signalscene 38 residualscene 38 active
30000.0590.03880.00warm-up0.0260warm-up
35000.0470.03730.000.0840.02350.00
40000.0450.03510.010.0770.02230.00
45000.0460.03051.000.0890.02130.00
50000.0370.02531.000.0970.02060.00
55000.0350.02301.000.0900.01990.00
60000.0420.02820.000.0910.02390.00
65000.0380.02441.000.1160.01970.00
70000.0370.02291.000.1080.02110.00
75000.0390.02241.000.1080.02020.00
80000.0330.01901.000.1110.01920.00
85000.0380.02081.000.1130.01940.00
90000.0350.01770.000.0970.01890.01
95000.0330.01820.000.0980.01831.00
100000.0380.01690.000.0900.01671.00

In scene 40 the normalized signal decreases together with the residual, and the weighting is activated early in training, interrupted around the opacity reset and then activated again. In scene 38 the residual also decreases, but the normalized signal increases, so the weighting is activated only late in training. These traces indicate that the normalized signal does not consistently track the absolute error and that its switching is affected by the opacity reset. We will therefore describe the controller as a heuristic that reacts to changes in the relative spatial distribution of reconstruction difficulty and sets the timing of reliability weighting, rather than as a detector of geometric convergence.

Separate ablation of each component

Table A3. Component ablation with our recipe on the same selected SEVA set. Values are the mean and standard deviation over three training seeds, averaged over all eleven scenes. The no pruning and random pruning rows use one training seed, and random pruning averages five random draws of the kept views. The last row is the per-scene standard deviation across training seeds, averaged over scenes and variants.

variantview pruningreliability weightingwhen weighting is appliedAll F1Unobs F1All Chamfer
no pruningnone, all 16 generated views keptnonenever0.7230.71432.38
random pruningrandom 14 of 16 viewsnonenever0.8130.81022.69
pruning onlyours, 14 of 16 viewsnonenever0.818 ± 0.0100.823 ± 0.01223.35 ± 2.32
always onours, 14 of 16 viewsfull, difficulty and consistencyfrom the first iteration0.820 ± 0.0020.819 ± 0.00222.34 ± 0.56
fixed scheduleours, 14 of 16 viewsfull, difficulty and consistencyfrom the end of warm-up onward0.821 ± 0.0050.818 ± 0.00222.30 ± 0.60
full methodours, 14 of 16 viewsfull, difficulty and consistencyadaptive controller0.822 ± 0.0100.823 ± 0.00822.08 ± 0.45
difficulty onlyours, 14 of 16 viewsdifficulty cue onlyadaptive controller0.817 ± 0.0080.818 ± 0.00722.26 ± 0.91
consistency onlyours, 14 of 16 viewsconsistency cue onlyadaptive controller0.816 ± 0.0050.818 ± 0.00222.46 ± 0.29
standard deviation across training seeds, per scene0.0190.0201.54

Table A3 analyzes the contribution of each component under the same generated views and the same recipe. Removing generated views has the largest effect, substantially improving both F1 and Chamfer over the no pruning variant. Random pruning of the same number of views performs at a similar level to our pruning criterion, also when compared at the same training seed, which indicates that the benefit comes mainly from limiting the generated supervision rather than from the specific views that are removed. Adding reliability weighting yields a slightly lower Chamfer, and the adaptive controller, the fixed schedule and always-on weighting perform comparably, as do the single-cue variants. The differences among the weighting and activation variants lie within the variation across training seeds.

We will revise the paper accordingly. The main claim will be that filtering generated views is necessary for our recipe. The pruning criterion, reliability weighting and adaptive controller will be presented as practical design choices with small measured effects, not as independent sources of accuracy.

We agree that OracleGS is the closest prior work. It uses generated observations, uncertainty-aware supervision and progressive augmentation. Our pipeline differs in four design choices. It selects one candidate sequence before reconstruction, prunes generated views before optimization, weights the remaining views patch-wise during optimization, and activates this weighting adaptively. As shown above, the measured benefit comes mainly from removing generated views, while the weighting and its activation have small effects.

We ran OracleGS from its official implementation in our common setting, as described in our answer below.

Tables A5 to A11 show that our method achieves a higher overall F1 and Observed F1 and a lower Chamfer than this OracleGS baseline, while OracleGS achieves a slightly higher Unobserved F1. These results indicate that our contribution lies in overall geometry and the preservation of observed surfaces, with unobserved-region accuracy comparable to OracleGS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.