acceptodds
Under review as a conference paper at ICLR 2027

HarmoGS: Robust 3D Gaussian Splatting in the Wild via Conflict-Aware Gradient Harmonization

19 views

Abstract

In-the-wild 3D Gaussian Splatting remains challenging due to transient distractors and illumination-induced cross-view appearance inconsistencies. Existing methods mainly rely on image-level masking to suppress unreliable supervision, but masking alone cannot fully eliminate residual occlusions or resolve illumination-induced inconsistencies, both of which can introduce conflicting cross-view gradients. These unresolved conflicts may destabilize Gaussian optimization and lead to visible reconstruction artifacts. We propose a conflict-aware 3DGS framework that addresses this problem from both image-space supervision and gradient-level optimization. Semantic Consistency-Guided Masking learns pixel-wise consistency scores to adaptively refine prior masks and suppress unreliable supervision before gradient formation. A dual-view Conflict-Aware Gradient Harmonization strategy further reconciles view-specific gradients by mutually rotating them into an orthogonal configuration, reducing negative directional interference across views. We also introduce conflict-aware densification and pruning to stabilize Gaussian growth and suppress persistently conflicting primitives. Extensive experiments on standard in-the-wild benchmarks demonstrate that our method achieves state-of-the-art rendering quality under complex transient distractors and cross-view inconsistencies.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Gaussian Splatting, novel view synthesis, in-the-wild reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6jNROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HarmoGS consists of three main parts:

  1. Mask guided by semantic consistency. Using frozen DINOv2 features to predict pixel reliability, and constructing training targets with semantic differences and photometric residuals; Subsequently, the predicted scores are fused with the binary prior provided by SAM to reduce the impact of unreliable regions on reconstruction loss.
  2. Dual view gradient coordination. Sample two views each time and calculate the reconstruction gradient separately. When the inner product of the gradient of a certain attribute group is negative, rotate the gradient in the plane formed by the two to make them orthogonal to each other, and then add them up for updating. The appendix clearly states that this step uses gradients aggregated by attribute groups for the entire scene, rather than Gaussian independent coordination.
  3. Enhancement and pruning of conflict perception. Use the reciprocal of the position gradient coordination coefficient to adjust the density statistics, and attenuate the opacity based on the cumulative collision between the Gaussian position and the opacity gradient. Finally, remove some Gaussian through conventional pruning. The method is based on Mip Splatting and validated on NeRF On the go, Robust NeRF, and PhotoTourism.
Critical strengths

1 The improvement of PSNR is common and has been observed in most scenarios. 2 The experimental analysis is quite comprehensive.

Critical weaknesses

1 Negative inner product itself can occur in normal multi view optimization. Figure 1 illustrates the widespread existence of conflicts, but it cannot be proven separately that these conflicts mainly come from transient objects or lighting changes, nor can it prove that all processed conflicts are harmful.

2 After gradient correction, there may be a situation where the direction of the synthesized gradient remains unchanged while the amplitude changes, which requires further explanation for the performance improvement brought by direction coordination.

Questions for authors

See Weaknesses and review comments.

5 comments

We sincerely thank the reviewer for the thoughtful evaluation of our work. We appreciate the reviewer’s acknowledgment of the performance improvements achieved by our method and the comprehensiveness of our experimental analysis. Below, we address the specific concerns.

1. Clarifying the Nature, Sources, and Impact of Cross-View Gradient Conflicts

Response: Thank you for raising this important concern. We agree that negative gradient inner products are not exclusive to in-the-wild reconstruction and may naturally arise in standard multi-view optimization. Importantly, our method does not assume that all negative inner products are harmful or necessarily originate from transient objects or illumination changes. Rather, we use them as indicators of directional disagreement between view-specific gradients and aim to mitigate potential destructive interference during optimization. To further clarify these issues, we provide the following analyses.

Cross-Dataset Analysis of Gradient Conflicts. To better characterize cross-view gradient conflicts, we first evaluate two standard benchmarks: NeRF Synthetic, which consists of synthetic scenes without complex natural illumination variations, and Mip-NeRF360, which contains real indoor and outdoor scenes captured under diverse natural lighting conditions. We define the conflict rate (CR) and conflict severity (S) as:

CR=1T∑t=1TI(ct<0)×100\mathrm{CR}=\frac{1}{T}\sum_{t=1}^{T}\mathbb{I}(c_t < 0)\times 100\\%,

S=1T∑t=1Tmax⁡(0,−ct),S=\frac{1}{T}\sum_{t=1}^{T}\max(0,-c_t),

where

ct=gA,t⊤gB,t∥gA,t∥2∥gB,t∥2=cos⁡θt.c_t=\frac{\mathbf{g}_{A,t}^{\top}\mathbf{g}_{B,t}}{\lVert\mathbf{g}_{A,t}\rVert_2\lVert\mathbf{g}_{B,t}\rVert_2}=\cos\theta_t.

Here, TT denotes the total number of evaluated gradient pairs, and a larger SS indicates more severe gradient conflicts.

As shown below, Mip-NeRF360 exhibits a higher conflict rate and greater conflict severity than NeRF Synthetic, suggesting that cross-view gradient disagreement becomes more pronounced in real scenes with more diverse appearance and illumination conditions. This trend is further amplified in the in-the-wild NeRF On-the-go dataset, where both the conflict rate and severity are substantially higher than those of the standard benchmarks. Notably, even after applying semantic consistency-guided masking, the conflict severity remains 2.63 times that of NeRF Synthetic, while the proportion of severe conflicts with θ>110∘\theta > 110^\circ remains 7.95 times higher. These results indicate that cross-view gradient conflicts are particularly pronounced in challenging in-the-wild scenes and cannot be fully eliminated by image-space masking alone.

Table: Cross-view gradient conflict statistics across different datasets.

DatasetSettingConflict RateConflict Severityθ>100∘\theta > 100^\circθ>110∘\theta > 110^\circ
NeRF SyntheticStandard (without complex natural illumination)32.51% (1.00$\times$)0.01554 (1.00$\times$)1.68% (1.00$\times$)0.37% (1.00$\times$)
Mip-NeRF360Standard (with diverse natural lighting)36.59% (1.13$\times$)0.02686 (1.73$\times$)4.48% (2.67$\times$)1.76% (4.76$\times$)
NeRF On-the-goIn-the-wild (with mask)43.66% (1.34$\times$)0.04092 (2.63$\times$)7.39% (4.40$\times$)2.94% (7.95$\times$)
NeRF On-the-goIn-the-wild (without mask)46.21% (1.42$\times$)0.04348 (2.80$\times$)8.31% (4.95$\times$)3.38% (9.14$\times$)

Controlled Analysis of Conflict Sources (Illumination and Occlusion). To investigate whether illumination variations and transient occlusions can contribute to gradient conflicts, we conduct a controlled experiment on TensoIR-Synthetic. We use its built-in illumination transformations and introduce randomly colored and shaped transient occluders into approximately 20% of the training images. As shown below, both factors increase conflict rate and severity, with their combination producing the strongest conflicts. This controlled experiment confirms that both factors can exacerbate gradient conflicts, while not establishing them as the dominant sources of conflicts in real-world scenes.

Table: Controlled analysis of illumination variations and transient occlusions on TensoIR-Synthetic.

ConditionIllumination VariationTransient OcclusionConflict Rate (%)Conflict Severity
Clean✗✗36.200.01056
Illumination only✓✗44.960.03844
Occlusion only✗✓43.930.03449
Illumination + Occlusion✓✓45.930.04472

(Continued in the next official comment due to length.)

Gradient Harmonization Yields Greater Gains in the Wild. We further evaluate the reconstruction benefits of gradient harmonization. As shown below, it brings only marginal improvements on standard benchmarks but substantially greater gains on NeRF On-the-go. This observation is consistent with the higher conflict severity in in-the-wild scenes, suggesting that gradient harmonization is particularly beneficial under substantial cross-view inconsistencies.

Table: Effect of Conflict-Aware Gradient Harmonization on standard and in-the-wild scene reconstruction.

Scene TypeDatasetFrameworkGradient HarmonizationPSNR↑SSIM↑LPIPS↓
StandardNeRF Synthetic3DGS✗33.320.96950.0303
StandardNeRF Synthetic3DGS✓33.350.96980.0302
StandardMip-NeRF3603DGS✗27.450.81560.2156
StandardMip-NeRF3603DGS✓27.540.81680.2151
In-the-wildNeRF On-the-goHarmoGS✗23.500.8020.131
In-the-wildNeRF On-the-goHarmoGS✓23.980.8180.123

All Conflicts vs. Selective Harmonization. Given that the previous analysis shows a higher proportion of large-angle conflicts in in-the-wild scenes, we further investigate whether harmonizing only these stronger conflicts is sufficient. As shown below, harmonizing all conflicting pairs achieves the best reconstruction quality, while all strategies incur virtually identical training time (0.33h). Given the negligible computational overhead, we therefore apply gradient harmonization to all negative-inner-product pairs without introducing additional thresholds.

Table: Effect of Selective Gradient Harmonization on Reconstruction Quality for NeRF On-the-go.

Harmonization StrategyCorrection CriterionPSNR ↑SSIM ↑LPIPS ↓Training Time (h) ↓
No harmonizationNone23.500.8020.1310.33
All conflictsθ>90∘\theta > 90^\circ23.980.8180.1230.33
Strong conflicts onlyθ>100∘\theta > 100^\circ23.760.8130.1260.33
Very strong conflicts onlyθ>110∘\theta > 110^\circ23.700.8120.1280.33

Overall, our experiments show that negative gradient inner products indicate directional disagreement rather than necessarily harmful interference. Such conflicts also occur in standard multi-view optimization but become more pronounced in challenging in-the-wild scenes, where illumination variations and transient occlusions further exacerbate them. Although not every negative inner product is necessarily harmful, harmonizing all conflicting pairs achieves better reconstruction quality than selective harmonization with negligible additional overhead. We will clarify these distinctions in the revised discussion of Fig. 1 to avoid overclaiming the causes and effects of individual conflicts.

2. Clarifying the Effect of Gradient Harmonization on the Aggregated Update

Response: We agree that gradient harmonization does not necessarily change the direction of the final aggregated gradient in every case. Importantly, our objective is not to enforce a change in the final update direction, but to adjust the relative directions of conflicting view-specific gradients, thereby eliminating their destructive interference.

For a given attribute group, consider two nonzero gradients with angle θ∈(π/2,π)\theta\in(\pi/2,\pi). Under our default norm-preserving symmetric correction (ρ=0.5\rho=0.5), the aggregated gradient retains its original direction if and only if the two gradients have equal magnitudes. In this case, writing ∥g1∥=∥g2∥=G\|\mathbf{g}_1\|=\|\mathbf{g}_2\|=G, the harmonized gradients remain symmetric around the original angular bisector. Their aggregate magnitude therefore increases from

=2G\cos\frac{\theta}{2}$$ to $$\|\widetilde{\mathbf{g}}_1+\widetilde{\mathbf{g}}_2\| =\sqrt{2}G,$$ since the harmonized gradients are orthogonal and preserve their individual magnitudes. To clarify the reduction in cancellation, expanding the squared norm of the original gradient sum gives $$\|\mathbf{g}_1+\mathbf{g}_2\|^2 =\|\mathbf{g}_1\|^2+\|\mathbf{g}_2\|^2 +2\mathbf{g}_1^\top\mathbf{g}_2.$$ The last term is negative because $\theta > \pi/2$, indicating destructive gradient cancellation. Our harmonization eliminates this negative cross-term by making the two gradients orthogonal, thereby preserving more effective optimization signals. The magnitude increase in this example is a consequence of eliminating destructive gradient cancellation, rather than an explicitly imposed amplification. Therefore, an unchanged aggregated direction does not imply ineffective coordination: the benefit of gradient harmonization is not necessarily reflected in a change in the final aggregated direction; it also lies in reducing destructive interference and preserving effective optimization signals from both views.

Thanks to the author's rebuttal. Some of my concerns have been addressed. I will consider adjusting my score after reading the author's revised paper.

Reviewer_9VD2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HarmoGS, a robust 3D Gaussian Splatting method for unconstrained real‑world scenes with dynamic distractors and lighting variations. It leverages semantic‑aware masking to filter noisy supervision and harmonizes conflicting cross‑view gradients to stabilize model updates. It also applies conflict‑driven densification and pruning to refine Gaussian primitives and reduce reconstruction artifacts. Evaluations on multiple real‑world datasets demonstrate its superior rendering performance against prior state‑of‑the‑art approaches.

Critical strengths
  1. The manuscript is well-structured and easy to follow. The individual technical components in each section are tightly coupled, successfully forming a cohesive and unified framework.
  2. The paper demonstrates strong theoretical grounding. The proposed method is supported by rigorous mathematical derivations and proofs, which provide solid theoretical backing for the core claims.
Critical weaknesses
  1. The performance improvement over the baseline (AsymGS) on certain datasets appears marginal. Furthermore, the evaluation lacks comparisons with more recent state-of-the-art methods, which limits the manuscript's ability to convincingly demonstrate the effectiveness of the proposed approach.
  2. According to Gradient-Direction-Aware Density Control for 3D Gaussian Splatting (ICLR 2026), gradient direction variance within the same Gaussian can naturally occur even in standard scenes due to pixel-coverage differences. Consequently, the core motivation of this paper regarding gradient conflict is debatable, unless the authors can provide a concrete comparison demonstrating the actual severity of gradient conflict between standard and in-the-wild (wild) scenes.
  3. 3D Gaussian parameters differ significantly in dimensionality and physical meaning. Applying Conflict-Aware Gradient Harmonization indiscriminately across all parameters is conceptually questionable. Since gradients for distinct attributes have vastly different magnitudes and physical optimization landscapes, harmonizing them uniformly may introduce unintended artifacts or sub-optimal optimization dynamics.
  4. The framework diagram contains too many formulas and text blocks, making it difficult to digest quickly.
Questions for authors
  1. The authors should include comparisons with more recent baselines, such as MetroGS (CVPR 2026), to better contextualize their performance improvements.
  2. The authors should provide further explanations or quantitative comparisons regarding their motivation. Specifically, they need to clarify why gradient conflict poses a particularly severe challenge in "in-the-wild" datasets compared to standard setups.
  3. The authors should conduct an ablation study to demonstrate that the proposed gradient conflict design is valid and beneficial across different learnable parameters (e.g., position, rotation, scale, and opacity).
4 comments

We sincerely thank the reviewer for the positive assessment of our work. We appreciate the reviewer’s recognition of the clear manuscript organization, the cohesive integration of our technical components, and the rigorous mathematical formulations and theoretical analysis. Below, we address the specific concerns.

1. Performance Improvements over AsymGS and Comparisons with more Recent State-of-the-Art Methods (such as MetroGS)

Response: Thank you for raising this point. While the magnitude of improvement varies across individual scenes, with relatively modest gains on some scenes, HarmoGS nevertheless consistently outperforms AsymGS across all 13 evaluated scenes, particularly in PSNR. This consistency indicates that the improvements are not limited to a few favorable scenes. Specifically, HarmoGS achieves an average PSNR gain of 0.37 dB on NeRF On-the-go, with scene-wise gains ranging from 0.11 to 0.48 dB on RobustNeRF and 0.45 to 0.80 dB on PhotoTourism. Importantly, these quantitative improvements are also reflected in the visual results. As shown in Figs. 5 and 7, HarmoGS more effectively suppresses floating and transient artifacts while preserving finer scene details, resulting in visibly improved reconstruction quality. Moreover, compared with AsymGS, HarmoGS achieves better reconstruction quality while reducing training time by 19.5% (0.41 h vs. 0.33 h) and the number of Gaussians by 37.0% (546K vs. 344K), demonstrating faster optimization and a more compact Gaussian representation.

Additional comparisons with more recent state-of-the-art methods. We appreciate this helpful suggestion. In addition to the comparisons with recent methods included in the original manuscript, we have further expanded our evaluation to include DualSplat (whose results were initially presented in the supplementary material), ForestSplats, PriSplat, WilLaGS, and MetroGS. Note that MetroGS is primarily designed for complex urban scene reconstruction rather than in-the-wild scenes, and does not natively incorporate a mask-based design. Therefore, we additionally reproduce MetroGS on the NeRF On-the-go dataset under both the original and mask-assisted settings (MetroGS and MetroGS + Mask) to provide a more comprehensive comparison. The additional quantitative results are reported below.

Table: Average quantitative comparisons with recent state-of-the-art methods on NeRF On-the-go.

MethodPSNR ↑SSIM ↑LPIPS ↓
DualSplat(CVPR 2026)23.110.7970.127
ForestSplats(WACV 2026)23.400.8070.156
PriSplat(ECCV 2026)23.110.7760.134
MetroGS (CVPR 2026)17.300.5980.328
MetroGS + mask20.730.7460.162
HarmoGS(Ours)23.980.8180.123

Table: Average quantitative comparisons with recent state-of-the-art methods on PhotoTourism.

MethodPSNR ↑SSIM ↑LPIPS ↓
WilLaGS(ECCV 2026)25.510.8830.151
ForestSplats(WACV 2026)25.360.8710.151
HarmoGS(Ours)26.210.8810.149

As shown above, HarmoGS achieves the best performance across all three metrics on NeRF On-the-go among the methods with available results. On PhotoTourism, HarmoGS achieves the best PSNR and LPIPS while maintaining competitive SSIM. These results further demonstrate the strong and consistent performance of HarmoGS across different datasets and evaluation metrics. We will incorporate these additional comparisons (including MetroGS) and the corresponding discussions into the revised manuscript and supplementary material.

2. Simplifying the Framework Diagram for Better Clarity

Response: We thank the reviewer for this helpful suggestion and agree that Fig. 2 is overly dense. In the revised manuscript, we will simplify Fig. 2 to highlight the three main stages of our framework: dual-view reconstruction with semantic consistency-guided masking, attribute-wise conflict-aware gradient harmonization, and conflict-aware Gaussian densification and pruning. Detailed mathematical expressions and explanatory text will be removed from the figure while being retained in the corresponding method subsections. This minor refinement will make the overall framework clearer and easier to follow without sacrificing technical details.

(Continued in the next official comment due to length.)

3. A Comparative Analysis of Gradient Conflicts in Standard and In-the-Wild Scenes

Response: Thank you for pointing out this relevant work. We agree that gradient-direction inconsistency can naturally arise in standard 3DGS optimization, including from pixel-wise subgradient aggregation within individual Gaussians, as studied in GDAGS [Zhou et al., 2026]. However, the two methods address different aspects of gradient inconsistency: GDAGS measures pixel-wise gradient coherence to guide Gaussian densification, whereas our method focuses on conflicts among view-specific gradients of Gaussian attributes and harmonizes them during joint multi-view optimization. Our motivation is not that gradient conflicts are unique to in-the-wild scenes. Rather, we investigate whether they become more frequent and severe under challenging in-the-wild conditions.

Cross-Dataset Comparison of Gradient Conflict Severity. To quantitatively investigate this, we compare cross-view gradient conflicts across NeRF Synthetic, Mip-NeRF360, and NeRF On-the-go using the conflict rate and conflict severity, as defined in our response to Reviewer 6jNR. As shown below, NeRF On-the-go exhibits substantially higher conflict rates, greater conflict severity, and larger proportions of severe conflicts than both standard benchmarks. Notably, this trend persists even after applying semantic consistency-guided masking. These results demonstrate that cross-view gradient conflicts are considerably more pronounced in challenging in-the-wild reconstruction.

Table: Cross-view gradient conflict statistics across different datasets.

DatasetSettingConflict RateConflict Severityθ>100∘\theta>100^\circθ>110∘\theta>110^\circ
NeRF SyntheticStandard (without complex natural illumination)32.51% (1.00×)0.01554 (1.00×)1.68% (1.00×)0.37% (1.00×)
Mip-NeRF360Standard (with diverse natural lighting)36.59% (1.13×)0.02686 (1.73×)4.48% (2.67×)1.76% (4.76×)
NeRF On-the-goIn-the-wild (with mask)43.66% (1.34×)0.04092 (2.63×)7.39% (4.40×)2.94% (7.95×)
NeRF On-the-goIn-the-wild (without mask)46.21% (1.42×)0.04348 (2.80×)8.31% (4.95×)3.38% (9.14×)

Gradient Harmonization Yields Greater Gains in the Wild. Furthermore, we evaluate the impact of gradient harmonization on reconstruction performance. As shown below, it brings only marginal improvements on standard benchmarks, whereas substantially greater improvements are observed on NeRF On-the-go. This is consistent with the higher conflict severity in in-the-wild scenes, suggesting that gradient harmonization is particularly beneficial under substantial cross-view inconsistencies.

Table: Effect of Conflict-Aware Gradient Harmonization on standard and in-the-wild scene reconstruction.

Scene TypeDatasetFrameworkGradient HarmonizationPSNR ↑SSIM ↑LPIPS ↓
StandardNeRF Synthetic3DGS✗33.320.96950.0303
StandardNeRF Synthetic3DGS✓33.350.96980.0302
StandardMip-NeRF3603DGS✗27.450.81560.2156
StandardMip-NeRF3603DGS✓27.540.81680.2151
In-the-wildNeRF On-the-goHarmoGS✗23.500.8020.131
In-the-wildNeRF On-the-goHarmoGS✓23.980.8180.123

Overall, we acknowledge that gradient-direction inconsistency is a general phenomenon in 3DGS, as also studied by GDAGS. Our results further demonstrate that cross-view gradient conflicts are considerably more severe in in-the-wild scenes and that addressing them brings greater reconstruction benefits under such challenging conditions. We will clarify the distinction between these two perspectives and strengthen the corresponding motivation in the revised manuscript.

(Continued in the next official comment due to length.)

4. Concern Regarding the Validity of Uniform Conflict-Aware Gradient Harmonization across Heterogeneous 3D Gaussian Attributes and the Need for Attribute-wise Ablation Studies

Response: Thank you for raising this important concern. We agree that indiscriminately harmonizing a single concatenated gradient vector across heterogeneous Gaussian attributes could lead to suboptimal optimization.

We would like to clarify that our implementation does not use a single shared gradient vector or global correction coefficients. Instead, as described in Section 3.3 and Appendix D.1, we perform gradient harmonization independently for each attribute group (a∈{x,r,s,α,c}a \in \{\mathbf{x},\mathbf{r},\mathbf{s},\alpha,\mathbf{c}\}). Specifically, we compute scene-level gradients and attribute-specific harmonization coefficients independently for each group, and apply the resulting coefficients only to the corresponding attributes. Therefore, no cross-attribute gradient conflicts are evaluated, and the different dimensionalities, gradient magnitudes, and optimization characteristics of heterogeneous attributes are accounted for through independent conflict estimation and correction. Furthermore, we introduce additional conflict-based attenuation specifically for geometric attributes ({x,r,s}\{\mathbf{x},\mathbf{r},\mathbf{s}\}), reflecting their distinct optimization characteristics.

Our existing results in Appendix D.1 support this design. Scene-wide per-attribute harmonization achieves 23.98 PSNR, 0.818 SSIM, and 0.123 LPIPS, outperforming scene-wide all-attribute harmonization (23.78 PSNR, 0.815 SSIM, and 0.130 LPIPS). This further supports that independently estimating gradient conflicts for different attributes is more effective than applying a shared harmonization strategy across all attributes.

Nevertheless, independently harmonizing different attributes does not necessarily imply that GH is equally beneficial for every attribute. Following the reviewer's suggestion, we conducted additional attribute-wise ablation experiments to further investigate the individual contributions of different attribute groups. The experimental configurations are summarized below:

Table: Ablation of gradient harmonization across different Gaussian attribute groups on NeRF On-the-go.

VariantPositionScaleRotationOpacitySH ColorPSNR↑SSIM↑LPIPS↓
w/o Gradient Harmonization✗✗✗✗✗23.500.8020.131
Full HarmoGS✓✓✓✓✓23.980.8180.123
Geometry-only GH✓✓✓✗✗23.770.8080.128
Opacity-and-SH-only GH✗✗✗✓✓23.670.8040.128
w/o Position GH✗✓✓✓✓23.930.8150.126
w/o Scale GH✓✗✓✓✓23.860.8150.126
w/o Rotation GH✓✓✗✓✓23.850.8170.125
w/o Opacity GH✓✓✓✗✓23.860.8160.127
w/o SH Color GH✓✓✓✓✗23.910.8160.128

Here, GH denotes Gradient Harmonization. The results show that removing GH from any individual attribute group consistently degrades reconstruction performance, although the magnitude of the benefit varies across attributes. Moreover, applying GH only to geometric attributes or to opacity and SH color yields inferior results compared with the full model. Overall, these results support the effectiveness of attribute-specific GH across heterogeneous Gaussian attributes. By independently estimating and correcting gradient conflicts within each attribute-specific parameter space, our method avoids indiscriminate harmonization across heterogeneous parameters while exploiting their complementary contributions to improve reconstruction quality.

I thank the authors for their response and helpful clarifications, which have addressed several of my concerns. However, I still feel that the novelty and performance gains are relatively modest. On balance, I will maintain my original score.

Reviewer_QrN8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HarmoGS to improve the robustness of 3D Gaussian Splatting in unconstrained real-world scenes. The paper argues that even after using masks to suppress transient distractors, different views may still produce conflicting gradients due to residual occlusions and illumination variations. To address this issue, the authors introduce Semantic Consistency-Guided Masking and detect cross-view gradient conflicts during dual-view training. Conflicting gradients are rotated into an orthogonal configuration before being jointly used to update the Gaussian representation. The conflict information is further incorporated into Gaussian densification and pruning. Experiments on NeRF On-the-Go, RobustNeRF, and PhotoTourism demonstrate consistent improvements over several strong baselines.

Critical strengths
  • The paper addresses cross-view inconsistency directly at the gradient optimization level rather than relying solely on image-space masks. Since substantial gradient conflicts can remain even after masking, the proposed dual-view gradient harmonization provides a more targeted way to handle this issue than transient masking alone.

  • The experimental validation is comprehensive.

Critical weaknesses
  • When the gradients from two views are strongly conflicting, the final combined update direction after orthogonalization is not guaranteed to be a descent direction for both original view-specific losses simultaneously.

  • The explanation of Semantic Consistency-Guided Masking appears relatively late in the method section, even though the masks are already used earlier in the formulation. This ordering makes the method slightly less convenient to follow.

Questions for authors

The paper treats negative gradient inner products as harmful cross-view conflicts. Is this assumption too strong? Negative inner products may also arise from normal stochastic optimization differences or valid viewpoint changes, rather than necessarily indicating transient distractors or unreliable supervision.

2 comments

We sincerely thank the reviewer. We appreciate the reviewer’s recognition of our gradient-level approach to addressing cross-view inconsistencies, its complementarity to image-space masking, and the comprehensiveness of our experimental validation. Below, we address the specific concerns.

1. The combined update is not guaranteed to be a descent direction for both original view-specific losses.

Response: We thank the reviewer for pointing out this important issue. We agree that under extreme conditions, e.g., when two gradients are strongly conflicting and have substantially different magnitudes, the combined update may fail to be a descent direction for both original view-specific losses simultaneously. Nevertheless, our goal is to mitigate cross-view gradient conflicts and facilitate more stable optimization, rather than guarantee per-view descent at every iteration. This is supported by our ablation results, where removing Gradient Harmonization reduces PSNR from 23.98 to 23.50 and SSIM from 0.818 to 0.802. We further clarify the theoretical properties of our method and provide additional empirical evidence below.

Theoretical guarantee for the joint objective. Importantly, although simultaneous descent of the individual view-specific losses is not guaranteed, our symmetric correction strategy (ρ=0.5\rho=0.5, used throughout our experiments) provides a first-order descent guarantee for the equal-weight joint objective, regardless of the relative magnitudes of the two nonzero, non-collinear conflicting gradients.

Let a=∥g1∥a=\|\mathbf{g}_1\|, b=∥g2∥b=\|\mathbf{g}_2\|, and β=(θ−π/2)/2\beta=(\theta-\pi/2)/2, where θ∈(π/2,π)\theta\in(\pi/2,\pi) is the angle between the conflicting gradients. Using the orthonormal basis {u1,u2}\{\mathbf{u}_1,\mathbf{u}_2\}, the combined harmonized gradient can be expressed as

=(a\cos\beta-b\sin\beta)\mathbf{u}_1 +(a\sin\beta+b\cos\beta)\mathbf{u}_2.$$ Consequently, we obtain $$\begin{aligned} (\mathbf{g}_1+\mathbf{g}_2)^\top\widetilde{\mathbf{g}} &=(a^2+b^2)\cos\beta-2ab\sin\beta\ &\geq 2ab(\cos\beta-\sin\beta)>0. \end{aligned}$$ The inequality follows from $a^2+b^2\geq 2ab$ and $0<\beta<\pi/4$. Therefore, under standard gradient descent with a sufficiently small learning rate, our symmetric harmonization guarantees a first-order decrease of the joint objective $\mathcal{L}_{\mathrm{joint}}=\mathcal{L}_1+\mathcal{L}_2$, regardless of the relative gradient magnitudes. **Empirical analysis of descent behavior.** We further evaluate the descent behavior of the final harmonized gradients on the NeRF On-the-go dataset. Specifically, for each iteration with conflicting gradients, we measure the alignment of the combined harmonized gradient, $\widetilde{\mathbf{g}}_t=\widetilde{\mathbf{g}}_{1,t}+\widetilde{\mathbf{g}}_{2,t}$, with the original view-specific gradients and report the corresponding descent frequencies: Table: Empirical descent frequencies of the combined harmonized updates on conflicting iterations of the NeRF On-the-go dataset. | Metric | Descent criterion | Frequency | |---|---|---:| | View-1 descent | $\mathbf{g}_1^\top\widetilde{\mathbf{g}}_t>0$ | 99.97% | | View-2 descent | $\mathbf{g}_2^\top\widetilde{\mathbf{g}}_t>0$ | 99.98% | | Simultaneous descent | Both conditions hold | 99.95% | | Joint-objective descent | $(\mathbf{g}_1+\mathbf{g}_2)^\top\widetilde{\mathbf{g}}_t>0$ | 100.00% | Overall, simultaneous descent of both view-specific losses is achieved in 99.95% of conflicting iterations, despite the lack of a worst-case guarantee. Meanwhile, the combined update remains a descent direction for the joint objective in all evaluated conflicting iterations (100.00%), consistent with our theoretical analysis. These results suggest that failures of simultaneous per-view descent are rare in practice and do not compromise the overall optimization behavior of our method. ### **2. Improving the Organization of the Method Section** **Response:** We thank the reviewer for pointing out this issue. We agree that introducing the masks $M_i$ in Eq. (2) before explaining their construction in Section 3.4 makes the method less straightforward to follow. To improve readability, we will make a minor organizational adjustment to the Method section by introducing Semantic Consistency-Guided Masking before the joint dual-view optimization formulation. The revised order will be: 1. Semantic Consistency-Guided Masking. 2. Joint Optimization of Cross-View Gaussian Parameters. 3. Conflict-Aware Gradient Harmonization. 4.Conflict-Aware Gaussian Densification and Pruning. This is a minor reorganization of the existing content that does not change the technical formulations or the proposed method. It simply clarifies how the masks are constructed before they are used in the reconstruction objectives, providing a more natural flow of the method. **(Continued in the next official comment due to length.)**

3. Clarifying Whether Negative Gradient Inner Products Indicate Harmful Conflicts

Response: Thank you for raising this important concern. We agree that negative gradient inner products do not necessarily indicate harmful conflicts, as they may also arise from stochastic optimization variations or valid viewpoint-dependent supervision. Importantly, our method does not assume that every negative inner product represents unreliable supervision or a harmful update. Instead, we use it as an indicator of directional disagreement between view-specific gradients, which may cause destructive interference when aggregated. Our goal is to mitigate such potential interference rather than discard potentially useful view-specific information.

Cross-Dataset Analysis of Gradient Conflicts. To investigate whether cross-view gradient conflicts are more pronounced in in-the-wild scenes, we compare three datasets: NeRF Synthetic (a standard 3DGS benchmark with synthetic scenes without complex natural illumination), Mip-NeRF360 (a standard 3DGS benchmark comprising real scenes under diverse natural lighting), and NeRF On-the-go (an in-the-wild dataset). We quantify gradient conflicts using the conflict rate and conflict severity, as defined in our response to Reviewer 6jNR. As shown below, both conflict rate and severity are higher on Mip-NeRF360 than on NeRF Synthetic, and are further elevated on NeRF On-the-go. Notably, even after semantic consistency-guided masking, NeRF On-the-go exhibits 1.52× higher conflict severity than Mip-NeRF360, along with a higher proportion of large-angle conflicts. These results indicate that cross-view gradient disagreements are more pronounced in challenging in-the-wild scenes.

Table: Cross-view gradient conflict statistics across different datasets.

DatasetSettingConflict RateConflict Severityθ>100∘\theta > 100^\circθ>110∘\theta > 110^\circ
NeRF SyntheticStandard (without complex natural illumination)32.51% (1.00$\times$)0.01554 (1.00$\times$)1.68% (1.00$\times$)0.37% (1.00$\times$)
Mip-NeRF360Standard (with diverse natural lighting)36.59% (1.13$\times$)0.02686 (1.73$\times$)4.48% (2.67$\times$)1.76% (4.76$\times$)
NeRF On-the-goIn-the-wild (with mask)43.66% (1.34$\times$)0.04092 (2.63$\times$)7.39% (4.40$\times$)2.94% (7.95$\times$)
NeRF On-the-goIn-the-wild (without mask)46.21% (1.42$\times$)0.04348 (2.80$\times$)8.31% (4.95$\times$)3.38% (9.14$\times$)

Effect of Selective Gradient Harmonization. Given the higher severity and larger-angle conflict proportions observed in in-the-wild scenes, we further investigate whether harmonizing only stronger conflicts is sufficient. We compare harmonizing all negative-inner-product pairs (θ>90∘\theta > 90^\circ) with selectively processing stronger conflicts (θ>100∘\theta > 100^\circ and θ>110∘\theta > 110^\circ). As shown below, harmonizing all conflicting pairs achieves the best reconstruction quality, while all strategies incur virtually identical training time (0.33h). This supports our choice of using negative inner products as a practical coordination criterion, without introducing additional severity thresholds.

Table: Effect of Selective Gradient Harmonization on Reconstruction Quality for NeRF On-the-go.

Harmonization StrategyCorrection CriterionPSNR ↑SSIM ↑LPIPS ↓Training Time (h) ↓
No harmonizationNone23.500.8020.1310.33
All conflictsθ>90∘\theta > 90^\circ23.980.8180.1230.33
Strong conflicts onlyθ>100∘\theta > 100^\circ23.760.8130.1260.33
Very strong conflicts onlyθ>110∘\theta > 110^\circ23.700.8120.1280.33

Overall, we do not claim that every negative inner product is harmful; rather, we use it as a practical indicator of potential gradient interference. Our cross-dataset analysis shows that such conflicts are more pronounced in in-the-wild scenes, while the controlled experiments on TensoIR-Synthetic (see our response to Reviewer 6jNR) further demonstrate that illumination variations and transient occlusions can exacerbate them. Moreover, harmonizing all conflicting pairs improves reconstruction quality with negligible computational overhead. We will clarify this distinction in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.