acceptodds
Withdrawn from ICLR 2027

PriorGuard-GS: Taming Overfitting in Sparse-View Reconstruction with Guidance from Pretrained Priors

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing scenes from sparse views requires fitting observed images while generalizing to novel views. We present PriorGuard-GS (PGGS), which uses pretrained feed-forward priors to tame overfitting in sparse-view reconstruction. Prior-Guided Latent Optimization (PLO) retains the pretrained prior during adaptation, implicitly regularizing scene updates through learned relationships among Gaussian attributes. Conditional Gaussian Decoding (CGD) generates a variable set of rendering Gaussians, allowing the representation to evolve during optimization. Dynamic Event Awareness (DEA) preserves state across densification, pruning, and opacity resets during optimization. This formulation provides a common interface for multiple feed-forward architectures. We evaluate PGGS across four datasets and five backbones. Across backbones, PGGS improves overall novel-view PSNR by 1.48–4.10 dB over direct Gaussian optimization from the same initialization. With DepthSplat, it outperforms all six optimization-based baselines in all three aggregate metrics. With its fastest backbone, PGGS requires only 30.2 s in the optimization loop on average, compared with 89.1–786.8 s for optimization-based baselines. Qualitative comparisons show reduced ghosting and fewer fragmented structures, supporting the role of continued prior guidance in limiting sparse-view overfitting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sparse-view reconstruction, 3D Gaussian Splatting, Novel view synthesis, Test-time optimization, Latent-space optimization, Pretrained priors

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MYumOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PriorGuard-GS for sparse-view scene reconstruction. Prior-Guided Latent Optimization updates a residual in an intermediate representation and replays a frozen pretrained decoder, so gradients pass through the learned mapping. Conditional Gaussian Decoding adds a variable set of rendering Gaussians conditioned on pretrained anchors with a fixed count and per-member local codes; the anchor attributes can still change during optimization. Dynamic Event Awareness carries state through densification, pruning, and opacity resets. The framework is evaluated with posed and unposed feed-forward reconstruction backbones, and its final Gaussian attributes can be exported without the decoder. The evaluation covers four scene datasets and five backbones, including a two-view stress setting.

Critical strengths
  1. PLO keeps the pretrained decoder active during scene adaptation, coupling changes across Gaussian attributes through the learned mapping instead of optimizing every decoded attribute independently.
  2. CGD keeps the number of pretrained anchors fixed while allowing the active rendering set to grow or shrink. This provides a clear interface for changing rendering capacity while preserving the original feed-forward prediction at initialization.
  3. DEA carries lineage, active-set, and opacity-reset information across decoder calls. This addresses a real state-management challenge when the rendering set changes during optimization.
Critical weaknesses
  1. The reported full configuration disables latent, code, and scale regularizers. Although the latent residual still passes through the frozen pretrained decoder, it has no explicit trust region, and the separate CGD branch adds scene-specific attribute residuals. The prior mapping therefore remains in the computation, but the extent to which it constrains the final optimized scene is unclear.
  2. New members can be added only under anchors whose accumulated position-gradient score crosses a threshold, with a cap of six active members per anchor and 15% of the anchor count per event. This may limit added capacity in underfit regions whose existing anchors receive weak gradients.
  3. If all members under an anchor are pruned, the anchor has no active rendering path and receives no direct rendering gradient. The paper does not describe how such an anchor could become active again, so recovery after complete pruning is unclear.
  4. When the selected source local code is exactly zero, its perturbation is also zero. The source and new member then receive the same event hint and identical non-opacity inputs to the conditional decoder, so the added member starts without a geometric or appearance offset and may initially duplicate its source.
Questions for authors
  1. With the latent, code, and scale regularizers disabled, what constrains scene-specific updates so that the frozen prior continues to shape the final attributes rather than mainly providing the decoder path and initialization?
  2. How can an underfit or fully pruned anchor regain rendering capacity if its position-gradient score does not cross the growth threshold?
  3. When a source local code is zero, what initially differentiates the new member from the source given the zero perturbation and shared event hint?
Reviewer_QN45OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses overfitting when sparse-view 3DGS is optimized from a feed-forward prediction. Instead of optimizing the predicted Gaussians directly, PriorGuard-GS (PGGS) keeps the pretrained decoder frozen in the loop and optimizes a residual on an intermediate latent (Prior-Guided Latent Optimization, PLO), so that attribute updates are shaped by the decoder's Jacobian. To support a variable number of Gaussians, a small scene-specific MLP generates Gaussians from the decoder outputs, which serve as anchors (CGD), and DEA keeps density-control events persistent across regenerations. On 24 scenes from four datasets, PGGS improves over the feed-forward prediction and over direct Gaussian optimization from the same initialization.

Critical strengths
  • A well-motivated use of the pretrained prior. Direct optimization of the predicted Gaussians uses the feed-forward prior only at initialization. Keeping the frozen decoder in the loop, so that updates to the Gaussians are coupled through the mapping learned across scenes, is a reasonable way to regularize sparse-view optimization, although its effectiveness is not yet clearly validated (see W1).
  • Applicability across feed-forward backbones. PGGS defines a common adaptation interface on an intermediate latent of the frozen model, and it is applied to five backbones with different architectures and latent representations, in both posed and unposed settings.
Critical weaknesses
  • The ablations do not isolate the contribution of continued prior guidance. The paper's central claim is that keeping the frozen decoder in the loop is what limits overfitting. In Table 5, however, PLO alone gives only a marginal PSNR gain over fixed-count direct optimization, and most of the full model's improvement appears only in the "+ CGD" row. That row changes two things at once: it adds MLP-generated residuals to each Gaussian, and it turns on densification, pruning and opacity resets (Table 18). In addition, no variant uses CGD without PLO. The improvement may therefore come from the anchor-plus-MLP parameterization or from density control rather than from the frozen prior. Two additional variants would resolve this: FF + CGD + DEA, i.e., the full model without PLO, in which the anchors stay at the feed-forward prediction; and FF + PLO + CGD with a fixed number of Gaussians and no density control. As all ablations use DepthSplat, repeating them on a second backbone would also help.

  • The reported margin over direct optimization is measured against a weakened baseline. The Direct baselines in Table 2 apply growth, pruning and opacity resets within 1,500 steps, without a growth limit. The authors' own ablation shows that direct optimization without these events is considerably better (App. C.5), meaning that density control under this schedule harms the direct baseline. On DepthSplat, PGGS improves on the Direct baseline of Table 2 by 4.10 dB PSNR averaged over input counts, the upper end of the 1.48–4.10 dB margin in the abstract, but on the fixed-count variant by only about 0.7 dB. Most of this margin therefore reflects how much density control harms direct optimization, rather than the benefit of PGGS. The fixed-count baseline should be reported for all backbones, together with the final number of Gaussians for each method.

  • The selection of the ACID and DL3DV scenes is not justified. While all nine Mip-NeRF 360 scenes and eight Tanks and Temples scenes are used, ACID and DL3DV are represented by only four and three scenes, respectively (Sec. 4.1), which are unlikely to represent these large and diverse datasets. This choice can affect the conclusions, since the relative performance of the methods differs across datasets (e.g., CoR-GS achieves the highest PSNR on ACID; Table 13), and DL3DV is one of only two datasets in the unposed evaluation (Table 3). The paper lists the scene identifiers (App. B.1) but not how they were chosen. The authors should state the selection criterion and, ideally, evaluate more scenes from these two datasets.

Questions for authors
  • The paper notes that PGGS fits the input views better than direct optimization (l. 321–323), which suggests that its gain does not come from fitting the inputs less. Could you report input-view and novel-view metrics for each Table 5 variant, so that the reduction in overfitting can be read directly from the gap between them?

  • In Table 5, adding CGD and DEA improves PSNR and SSIM but worsens LPIPS relative to FF + PLO. What causes this trade-off? Since the objective (Eq. 11) has no perceptual term, while backbones such as TokenGS were trained with LPIPS (App. A.1), would adding one change it?

Reviewer_JNKtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PriorGuard-GS (PGGS) proposes a sparse-view 3D Gaussian Splatting framework that reduces overfitting by preserving pretrained priors during scene-specific optimization. It combines latent-space refinement, adaptive Gaussian generation, and dynamic density control to improve reconstruction quality while maintaining prior guidance. Experiments across four datasets and five feed-forward backbones demonstrate improvements in novel-view synthesis over direct Gaussian optimization.

Critical strengths

The paper addresses a relevant overfitting problem in sparse-view reconstruction. Evaluations across five backbones demonstrate the framework's applicability, and component ablations provide evidence for the benefits of prior-guided optimization. The improvements in novel-view quality are encouraging, particularly under sparse input conditions.

Critical weaknesses
  1. Limited novelty of individual components. Latent-space test-time optimization and anchor-conditioned Gaussian decoding have been explored in existing methods, including TokenGS and Scaffold-GS. The contribution primarily lies in their integration with persistent dynamic event handling. The paper should better distinguish its technical contributions from these existing approaches.

  2. Efficiency comparisons are not sufficiently controlled. The reported optimization times vary substantially across backbones. For example, DepthSplat + PGGS requires 326.5 seconds, compared with 29.5 seconds for direct optimization using the same backbone. It feels like the claimed efficiency advantage relies on selecting faster backbones and comparing against optimization-based methods with different reconstruction pipelines.

  3. Limited evaluation robustness. The experiments use custom sparse-view splits with only three selected novel views per scene, and each setting is evaluated with a single optimization run.

Questions for authors
  1. Novelty: Could the authors clarify the key technical contributions beyond existing latent-space test-time optimization and anchor-conditioned Gaussian decoding methods, such as TokenGS and Scaffold-GS? In particular, what distinguishes the proposed dynamic event handling from conventional Gaussian densification and pruning?

  2. Efficiency: Given that PGGS introduces substantial optimization overhead compared with direct optimization using the same backbone, could the authors provide a more controlled efficiency comparison, including end-to-end runtime and memory consumption?

  3. Evaluation protocol: Could the authors additionally evaluate PGGS using established sparse-view protocols from FSGS or SparseGS, with standard held-out test views, to demonstrate generalization beyond the custom splits with only three novel views per scene?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.