acceptodds
Under review as a conference paper at ICLR 2027

CoReSplat: Consensus-Relative Residual Compensation for Distractor-Free 3D Gaussian Splatting

Abstract

Distractor-free 3D Gaussian Splatting (3DGS) aims to recover a static scene from real-world image collections containing dynamic distractors, which introduce inconsistent observations that can corrupt scene optimization and produce rendering artifacts. Existing methods mitigate distractors through reliability-aware supervision or explicit scene decomposition. However, photometric residuals induced by retained observations may still reflect both under-explained scene content and view-specific discrepancies, leaving it unclear how these residuals should guide shared-scene updates. To address this supervision-to-update ambiguity, we propose CoReSplat, a consensus-relative residual compensation framework that leverages cross-view consensus over residual histories to identify disagreement for training-time compensation. Specifically, CoReSplat maintains a per-view residual history to accumulate cross-iteration evidence of residual tendencies. It then aggregates geometrically corresponding residual histories into a cross-view consensus reference that captures their agreement across views. The departure of each view’s residual history from this reference defines its consensus-relative disagreement, which is applied as a stop-gradient additive correction to the current prediction before backpropagation. In this way, CoReSplat dynamically reshapes the effective photometric residual relative to the cross-view consensus, while reducing the immediate influence of view-specific discrepancies on scene updates. Experiments on two real-world benchmarks demonstrate improved novel-view synthesis and consistent performance gains when CoReSplat is integrated into representative distractor-free 3DGS baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, Novel View Synthesis, Distractor-Free Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JH1fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoReSplat enhances the distractor-free 3DGS technique by accumulating historical data on signed residuals across views, constructing geometrically aligned cross-view medians, and applying historical consistency and inconsistency as independent correction terms before calculating the reconstruction loss. The evaluation included two benchmark tests, as well as integration with DeSplat, RobustSplat, and DualSplat.

Critical strengths
  1. This correction method employs a modular design and does not introduce dependencies during the inference process.

  2. Table 3 shows that PSNR improved across all 33 combinations of methods and scenarios.

  3. The results of the ablation experiments confirm the advantages of methods such as consensus-based correction, time accumulation, and supervised weighted history.

Critical weaknesses
  1. Compared with related optimization methods (particularly HarmoGS), its added value has not yet been fully demonstrated.

  2. Rough and cumulative residual historical data may lead to unreliable consensus, this limitation has not yet been thoroughly verified.

  3. Explanations regarding computational complexity and additional training costs remain insufficient.

Questions for authors
  1. Is it possible to compare it with HarmoGS under the same settings?

  2. How reliable is this consensus mechanism? In static scenarios, can the compensation process preserve useful gradient information?

  3. How much training time and memory overhead does CoReSplat incur?

Reviewer_D34COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles the "supervision-to-update ambiguity" in distractor-free 3D Gaussian Splatting. The authors point out that while prior methods filter or downweight dynamic distractors via mask prediction, uncertainty estimation, or scene decomposition, retained observations still yield photometric residuals that blend unresolved static scene geometry with view-specific transient discrepancies.

Critical strengths

The paper cleanly isolates the supervision-to-update ambiguity—demonstrating that retaining an observation does not resolve how its photometric discrepancy should guide static 3D updates. Formulating this as a consensus-relative residual reshaping problem provides an insightful conceptual perspective that moves beyond standard binary mask suppression or explicit scene decomposition.

The framework acts purely as an optimization-time regularizer without altering native 3DGS primitive representations, loss formulations, or inference speeds. Demonstrating consistent, non-negative PSNR improvements when layered atop three distinct architectures across two standard benchmarks shows integration flexibility.

Critical weaknesses

The framework requires an extra forward rendering pass every iteration to compute post-update residuals, alongside cross-view depth-consistency checks and reprojection. The paper omits wall-clock optimization runtime, iteration throughput, and peak VRAM metrics, making it impossible to determine whether the modest metric improvements justify the added compute burden.

Bilinearly upsampling an extremely coarse 16 x 16 grid introduces screen-space bleeding across sharp depth discontinuities. Furthermore, delaying activation until iteration 20,000 indicates the mechanism cannot tolerate geometric noise or floaters during early densification, functioning primarily as a late-stage image-space dampener rather than a robust geometric optimization tool.

Empirical gains are modest (frequently < 0.1 dB PSNR, with flat or regressing SSIM) and only demonstrated as a late-stage add-on after 20k iterations of baseline training.

Questions for authors

What is the exact impact on training efficiency—specifically wall-clock training duration (in minutes/hours), training iterations per second, and peak VRAM consumption—when integrating CoReSplat into the baseline architectures on standard benchmark scenes? How does reconstruction quality vary across grid resolutions {8, 16, 32, 64}, and can you provide localized error maps at sharp occlusion boundaries to demonstrate that bilinearly upsampling the coarse 16x16 disagreement map does not cause residual bleeding across significant depth discontinuities?

Reviewer_PXjAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoReSplat proposes a training-time residual compensation module for distractor-free 3D Gaussian Splatting. It accumulates supervision-weighted signed residual histories for each training view, establishes cross-view correspondences using the current scene geometry, and constructs a consensus reference through median aggregation. The difference between a view’s residual history and this consensus is added to the rendered image with stopped gradients before evaluating the underlying reconstruction loss. All auxiliary states are discarded at inference. Experiments integrate the module into DeSplat, RobustSplat, and DualSplat on NeRF On-the-go and RobustNeRF, reporting consistent scene-wise PSNR improvements and additional component ablations.

Critical strengths
  1. The cross-baseline evaluation provides meaningful evidence of practical usefulness. CoReSplat improves PSNR on every evaluated method–scene combination across three underlying methods and two benchmarks. This consistency supports the module’s usefulness beyond a single reconstruction pipeline, even though the average gains are generally modest.
  2. The design introduces no additional inference-time requirements. Residual histories and consensus computation are used only during training, while inference uses the underlying renderer directly. The ablations further show that the complete consensus-relative formulation outperforms direct-history compensation and latest-only residual evidence, providing initial support for combining temporal and cross-view information.
Critical weaknesses
  1. The central interpretation of residual compensation is not sufficiently validated. The method replaces the effective residual with ev−Hv+Cve_v-H_v+C_v. When the current residual resembles its history, the update is driven approximately by the cross-view consensus. This can suppress residual distractors, but can also suppress legitimate errors associated with static content visible in only a few views or with view-dependent appearance. The paper appropriately acknowledges that disagreement is not a transient label, yet the experiments do not establish that the compensation preferentially removes nuisance discrepancies while preserving useful supervision. Aggregate image metrics cannot determine whether improvements are accompanied by degradation in reliable static regions, limiting support for the claimed resolution of supervision-to-update ambiguity.
  2. The role and validity of geometry-aligned consensus remain insufficiently isolated. Residual histories are averaged into a coarse 16×16 grid, while correspondence is established using a depth value at each cell center. A cell may span multiple surfaces and occlusion boundaries, so correspondence between cell centers does not ensure correspondence between the pooled residuals. The reported ablations show that consensus improves over direct-history compensation, but do not distinguish geometry-aligned residual reasoning from simpler coarse appearance correction. This distinction is consequential because the latter could also explain part of the reported gains without validating the proposed interpretation.
Questions for authors
  1. Targeted validation of the supervision trade-off: Is there evidence separating reconstruction quality in distractor-affected regions from reliably observed static regions? Does compensation remain beneficial on clean observations, particularly for details visible in relatively few views?
  2. Targeted validation of geometric consensus: How does geometry-aligned consensus compare with a simple per-view or coarse regional color-bias correction using the same residual histories? A control with disrupted cross-view correspondences would also help establish whether accurate alignment is essential to the gains.
  3. Clarification of correspondence and computational cost: What exact rendering-derived depth statistic is used at grid centers and reprojected locations? How sensitive are results to the 16×16 grid resolution, and what training-time overhead is introduced by post-update re-rendering and consensus construction?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.