- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HarmoGS consists of three main parts:
- Mask guided by semantic consistency. Using frozen DINOv2 features to predict pixel reliability, and constructing training targets with semantic differences and photometric residuals; Subsequently, the predicted scores are fused with the binary prior provided by SAM to reduce the impact of unreliable regions on reconstruction loss.
- Dual view gradient coordination. Sample two views each time and calculate the reconstruction gradient separately. When the inner product of the gradient of a certain attribute group is negative, rotate the gradient in the plane formed by the two to make them orthogonal to each other, and then add them up for updating. The appendix clearly states that this step uses gradients aggregated by attribute groups for the entire scene, rather than Gaussian independent coordination.
- Enhancement and pruning of conflict perception. Use the reciprocal of the position gradient coordination coefficient to adjust the density statistics, and attenuate the opacity based on the cumulative collision between the Gaussian position and the opacity gradient. Finally, remove some Gaussian through conventional pruning. The method is based on Mip Splatting and validated on NeRF On the go, Robust NeRF, and PhotoTourism.
Critical strengths
1 The improvement of PSNR is common and has been observed in most scenarios. 2 The experimental analysis is quite comprehensive.
Critical weaknesses
1 Negative inner product itself can occur in normal multi view optimization. Figure 1 illustrates the widespread existence of conflicts, but it cannot be proven separately that these conflicts mainly come from transient objects or lighting changes, nor can it prove that all processed conflicts are harmful.
2 After gradient correction, there may be a situation where the direction of the synthesized gradient remains unchanged while the amplitude changes, which requires further explanation for the performance improvement brought by direction coordination.
Questions for authors
See Weaknesses and review comments.
5 comments
Gradient Harmonization Yields Greater Gains in the Wild. We further evaluate the reconstruction benefits of gradient harmonization. As shown below, it brings only marginal improvements on standard benchmarks but substantially greater gains on NeRF On-the-go. This observation is consistent with the higher conflict severity in in-the-wild scenes, suggesting that gradient harmonization is particularly beneficial under substantial cross-view inconsistencies.
Table: Effect of Conflict-Aware Gradient Harmonization on standard and in-the-wild scene reconstruction.
| Scene Type | Dataset | Framework | Gradient Harmonization | PSNR↑ | SSIM↑ | LPIPS↓ |
|---|---|---|---|---|---|---|
| Standard | NeRF Synthetic | 3DGS | ✗ | 33.32 | 0.9695 | 0.0303 |
| Standard | NeRF Synthetic | 3DGS | ✓ | 33.35 | 0.9698 | 0.0302 |
| Standard | Mip-NeRF360 | 3DGS | ✗ | 27.45 | 0.8156 | 0.2156 |
| Standard | Mip-NeRF360 | 3DGS | ✓ | 27.54 | 0.8168 | 0.2151 |
| In-the-wild | NeRF On-the-go | HarmoGS | ✗ | 23.50 | 0.802 | 0.131 |
| In-the-wild | NeRF On-the-go | HarmoGS | ✓ | 23.98 | 0.818 | 0.123 |
All Conflicts vs. Selective Harmonization. Given that the previous analysis shows a higher proportion of large-angle conflicts in in-the-wild scenes, we further investigate whether harmonizing only these stronger conflicts is sufficient. As shown below, harmonizing all conflicting pairs achieves the best reconstruction quality, while all strategies incur virtually identical training time (0.33h). Given the negligible computational overhead, we therefore apply gradient harmonization to all negative-inner-product pairs without introducing additional thresholds.
Table: Effect of Selective Gradient Harmonization on Reconstruction Quality for NeRF On-the-go.
| Harmonization Strategy | Correction Criterion | PSNR ↑ | SSIM ↑ | LPIPS ↓ | Training Time (h) ↓ |
|---|---|---|---|---|---|
| No harmonization | None | 23.50 | 0.802 | 0.131 | 0.33 |
| All conflicts | 23.98 | 0.818 | 0.123 | 0.33 | |
| Strong conflicts only | 23.76 | 0.813 | 0.126 | 0.33 | |
| Very strong conflicts only | 23.70 | 0.812 | 0.128 | 0.33 |
Overall, our experiments show that negative gradient inner products indicate directional disagreement rather than necessarily harmful interference. Such conflicts also occur in standard multi-view optimization but become more pronounced in challenging in-the-wild scenes, where illumination variations and transient occlusions further exacerbate them. Although not every negative inner product is necessarily harmful, harmonizing all conflicting pairs achieves better reconstruction quality than selective harmonization with negligible additional overhead. We will clarify these distinctions in the revised discussion of Fig. 1 to avoid overclaiming the causes and effects of individual conflicts.
2. Clarifying the Effect of Gradient Harmonization on the Aggregated Update
Response: We agree that gradient harmonization does not necessarily change the direction of the final aggregated gradient in every case. Importantly, our objective is not to enforce a change in the final update direction, but to adjust the relative directions of conflicting view-specific gradients, thereby eliminating their destructive interference.
For a given attribute group, consider two nonzero gradients with angle . Under our default norm-preserving symmetric correction (), the aggregated gradient retains its original direction if and only if the two gradients have equal magnitudes. In this case, writing , the harmonized gradients remain symmetric around the original angular bisector. Their aggregate magnitude therefore increases from
=2G\cos\frac{\theta}{2}$$ to $$\|\widetilde{\mathbf{g}}_1+\widetilde{\mathbf{g}}_2\| =\sqrt{2}G,$$ since the harmonized gradients are orthogonal and preserve their individual magnitudes. To clarify the reduction in cancellation, expanding the squared norm of the original gradient sum gives $$\|\mathbf{g}_1+\mathbf{g}_2\|^2 =\|\mathbf{g}_1\|^2+\|\mathbf{g}_2\|^2 +2\mathbf{g}_1^\top\mathbf{g}_2.$$ The last term is negative because $\theta > \pi/2$, indicating destructive gradient cancellation. Our harmonization eliminates this negative cross-term by making the two gradients orthogonal, thereby preserving more effective optimization signals. The magnitude increase in this example is a consequence of eliminating destructive gradient cancellation, rather than an explicitly imposed amplification. Therefore, an unchanged aggregated direction does not imply ineffective coordination: the benefit of gradient harmonization is not necessarily reflected in a change in the final aggregated direction; it also lies in reducing destructive interference and preserving effective optimization signals from both views.Thanks to the author's rebuttal. Some of my concerns have been addressed. I will consider adjusting my score after reading the author's revised paper.
We sincerely thank the reviewer for the thoughtful evaluation of our work. We appreciate the reviewer’s acknowledgment of the performance improvements achieved by our method and the comprehensiveness of our experimental analysis. Below, we address the specific concerns.
1. Clarifying the Nature, Sources, and Impact of Cross-View Gradient Conflicts
Response: Thank you for raising this important concern. We agree that negative gradient inner products are not exclusive to in-the-wild reconstruction and may naturally arise in standard multi-view optimization. Importantly, our method does not assume that all negative inner products are harmful or necessarily originate from transient objects or illumination changes. Rather, we use them as indicators of directional disagreement between view-specific gradients and aim to mitigate potential destructive interference during optimization. To further clarify these issues, we provide the following analyses.
Cross-Dataset Analysis of Gradient Conflicts. To better characterize cross-view gradient conflicts, we first evaluate two standard benchmarks: NeRF Synthetic, which consists of synthetic scenes without complex natural illumination variations, and Mip-NeRF360, which contains real indoor and outdoor scenes captured under diverse natural lighting conditions. We define the conflict rate (CR) and conflict severity (S) as:
CR=T1∑t=1TI(ct<0)×100
S=T1∑t=1Tmax(0,−ct),
where
ct=∥gA,t∥2∥gB,t∥2gA,t⊤gB,t=cosθt.
Here, T denotes the total number of evaluated gradient pairs, and a larger S indicates more severe gradient conflicts.
As shown below, Mip-NeRF360 exhibits a higher conflict rate and greater conflict severity than NeRF Synthetic, suggesting that cross-view gradient disagreement becomes more pronounced in real scenes with more diverse appearance and illumination conditions. This trend is further amplified in the in-the-wild NeRF On-the-go dataset, where both the conflict rate and severity are substantially higher than those of the standard benchmarks. Notably, even after applying semantic consistency-guided masking, the conflict severity remains 2.63 times that of NeRF Synthetic, while the proportion of severe conflicts with θ>110∘ remains 7.95 times higher. These results indicate that cross-view gradient conflicts are particularly pronounced in challenging in-the-wild scenes and cannot be fully eliminated by image-space masking alone.
Table: Cross-view gradient conflict statistics across different datasets.
Controlled Analysis of Conflict Sources (Illumination and Occlusion). To investigate whether illumination variations and transient occlusions can contribute to gradient conflicts, we conduct a controlled experiment on TensoIR-Synthetic. We use its built-in illumination transformations and introduce randomly colored and shaped transient occluders into approximately 20% of the training images. As shown below, both factors increase conflict rate and severity, with their combination producing the strongest conflicts. This controlled experiment confirms that both factors can exacerbate gradient conflicts, while not establishing them as the dominant sources of conflicts in real-world scenes.
Table: Controlled analysis of illumination variations and transient occlusions on TensoIR-Synthetic.
(Continued in the next official comment due to length.)