- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a feedforward method to git 2D Gaussians to images, without the tedious and slow optimization process. It uses a U-Net to predict 3 Gaussian attributes: density, log-covariance, and color. Then it uses mass-equal Voronoi to map the smooth density (due to the supervision) to proper Gaussian centers. Finally, a shared refinement network updates the Gaussian attributes using rendering residuals. Experiments show the effectiveness and fast fitting speed of the proposed method.
Critical strengths
Overall, the paper is well-written. Strengths:
- comprehensive experiments show this method is both faster and producing higher quality than previous gaussian image fitting methods.
- the use of classical Voronoi is elegant and an intuitive way to derive Gaussian centers from the smoothly-predicted density map. And the authors provide a fast implementation for it.
- training a feedforward network to directly predict Gaussians is a good way to avoid slow optimization-based fitting, which has been a trend in 3DGS. Here the authors follow the same principle and greatly improve the quality and speed of fitting 2DGS for an image, which is a meaningful contribution to this field.
Critical weaknesses
- The practical use of this work need to be explained in more detail. From the paper it seems using 2D Gaussians to fit an image does not decrease the storage size. In such case this technology's benefit remains unclear.
- Evaluation coverage is limited. The main results use 512 x 512 crops from two relatively small benchmarks. Does the method perform well on high-resolution images? And how does it perform on in-the-wild images?
Questions for authors
-
Can the predicted Gaussians be integrated into an existing codec or downstream task to demonstrate an end-to-end benefit? For example from the perspective of compression, could you report actual bitrate–distortion results after quantization and coding, together with encoding and decoding costs?
-
Can the authors evaluate their method on more diverse datasets, including maybe internet photos, to show that the method can actually generalize beyond the training distribution?
4 comments
In-the-wild and high-resolution images. Following the reviewer's suggestion, we add four datasets of in-the-wild images, including internet photos, at 512×512 (protocol of Table 1): COCO val2017 (Lin et al., 2014), CLIC2020 (Toderici et al., 2020), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017). Gaussifier predicts how many Gaussians each image needs, while each baseline uses its own count. For a comparison at equal Gaussian count, we also run Gaussifier with exactly as many Gaussians as the baseline used on each image (the "Ours @" columns). At equal count, Gaussifier leads every baseline in the four new datasets, by 6.4–8.1 dB over AIR and Instant-GI and by 14.8–17.3 dB over Fast-2DGS, so the gain does not come from using more Gaussians.
| Dataset (PSNR, dB) | Ours (predicted N) | AIR | Ours @ AIR's N | Instant-GI | Ours @ Instant-GI's N | Fast-2DGS (70k) | Ours @ 70k |
|---|---|---|---|---|---|---|---|
| COCO val2017 (500 internet photos) | 40.69 | 33.42 | 40.29 | 31.60 | 38.72 | 27.63 | 43.12 |
| CLIC2020 (41 professional photos) | 39.93 | 30.88 | 37.91 | 30.00 | 36.36 | 26.68 | 41.46 |
| Urban100 (100, buildings) | 39.85 | 28.01 | 36.06 | 27.53 | 34.42 | 22.59 | 39.90 |
| Manga109 (109, drawings) | 39.16 | 28.84 | 36.25 | 27.27 | 34.98 | 23.05 | 39.36 |
We also add full-resolution frames up to 2K. Since Fast-2DGS cannot choose its own count and 70k (set for 512×512) is small at 2K, we run it with Gaussifier's per-image count. Gaussifier stays above 40.7 dB on all three datasets, and at equal Gaussian count it still leads AIR and Instant-GI by 5.8–7.8 dB and Fast-2DGS by 14.6–17.0 dB.
| Dataset (PSNR, dB) | Ours (predicted N) | AIR | Ours @ AIR's N | Instant-GI | Ours @ Instant-GI's N | Fast-2DGS @ Ours' N |
|---|---|---|---|---|---|---|
| Kodak-24 (768×512) | 40.78 | 32.17 | 39.94 | 30.67 | 37.88 | 25.65 |
| DIV2K valid (100, longest side 2040) | 40.91 | 33.23 | 40.02 | 32.06 | 38.56 | 23.94 |
| CLIC2020 (41, longest side 2048) | 40.74 | 34.41 | 40.22 | 32.81 | 38.60 | 26.18 |
The reviewer is welcome to try Gaussifier on their photos in the browser, using the WebGPU demo at the anonymous link in our reproducibility statement.
Thank you for the additional experiments. The codec integration and broader evaluations address my main concerns. While wider downstream benefits remain unverified, I find the gains in Gaussian decomposition quality and speed meaningful. Please clarify the conditions of the real-time claim. I will maintain my score.
We thank Reviewer o2K3 for the follow-up and for finding the gains in quality and speed meaningful.
Conditions of the real-time claim. The paper reports about 14 ms per 512×512 image (73/68 FPS on Kodak-24/DIV2K) on one RTX 5090. This uses FP16 with TensorRT/AOTInductor compilation and the adaptive Gaussian count, at about 40 dB PSNR. The time covers the full pipeline from the input image to the final Gaussians, measured after warm-up passes. At 1280×720 the paper reports 20 FPS.
Faster implementation. Since submission, we further removed bottlenecks, fused the update kernels, froze the group normalization and captured the pipeline with CUDA graphs. The model is about 4× faster than the submitted implementation and recovered some precision loss (enables shorter refinement). The table reports the new timings on L40S and RTX 5090, with 5 and 7 refinement steps.
L40S (sm89):
| Resolution | K=5 Latency (ms) | K=5 FPS | K=5 PSNR (dB) | K=7 Latency (ms) | K=7 FPS | K=7 PSNR (dB) |
|---|---|---|---|---|---|---|
| 512×512 | 3.4 | 292 | 40.23 | 4.6 | 216 | 40.82 |
| 768×512 | 4.9 | 206 | 40.11 | 6.7 | 150 | 40.82 |
| 1280×720 | 12.9 | 77 | 40.61 | 18.7 | 54 | 41.28 |
| 1920×1080 | 30.7 | 33 | 41.06 | 43.4 | 23 | 41.65 |
RTX 5090 (sm120):
| Resolution | K=5 Latency (ms) | K=5 FPS | K=5 PSNR (dB) | K=7 Latency (ms) | K=7 FPS | K=7 PSNR (dB) |
|---|---|---|---|---|---|---|
| 512×512 | 2.3 | 431 | 40.23 | 3.1 | 321 | 40.82 |
| 768×512 | 3.1 | 318 | 40.11 | 4.3 | 234 | 40.82 |
| 1280×720 | 7.9 | 126 | 40.61 | 11.1 | 90 | 41.28 |
| 1920×1080 | 17.5 | 57 | 41.06 | 24.5 | 41 | 41.66 |
Gaussifier now runs >30 FPS at 1080p at >40 PSNR at one L40S (57 FPS at one RTX 5090). At the current processing speed, Gaussifier becomes a promising tool as a on-the-fly real-time processing tool for training, inference and other graphic applications. We will state these conditions in the revised manuscript and update the numbers.
The updated implementation is in the anonymous repository (https://github.com/Gaussifier/gaussifier).
We thank the reviewer for the positive assessment and the concrete suggestions.
Practical benefit and storage size. 2D Gaussian image compression depends mainly on the representation itself and its codec (quantization, entropy coding, etc.). How the Gaussians are produced, either by per-image fitting or feedforward, has limited effect on compression. Gaussifier targets a different problem that is orthogonal to compression.
The benefit we claim is speed and quality in producing Gaussian images.
The count is learned, and the user can set it. The predicted 42.6k–72.2k Gaussian count is learned from our per-image fit data which reaches a median of 42.3 dB PSNR: it targets fidelity rather than compactness. Additionally, users can set the count, e.g., 20k in Table 1 or 1k to 10k in the codec study below.
Integration into the GaussianImage codec. In the study below, we follow GaussianImage's pipeline but replace its image overfitting stage with Gaussifier's prediction. We then apply GaussianImage's codec: FP16 positions, 6-bit Cholesky factors, 2-stage residual vector quantization of colors, quantization-aware fine-tuning, and entropy coding. We run both Gaussifier and GaussianImage at six Gaussian counts from 1k to 10k (0.23–2.1 bpp), and every rate is a real file size. BD-rate is the percentage of extra bits Gaussifier needs relative to GaussianImage for the same PSNR, averaged over the PSNR range the two methods share (the lower the better).
Per 512×512 image near 1 bpp on one L40S, encoding takes 48 s for GaussianImage and 1.8 s for Gaussifier, and both decode in 1.1 ms. Both encoding times include GaussianImage's quantization-aware fine-tuning stage, which needs 7k instead of 50k iterations when starting from Gaussifier. Gaussifier thus makes GaussianImage about 27× faster to encode, by replacing its costly image overfitting stage and shortening its fine-tuning. In addition, Gaussifier needs 6.5%/3.4% fewer bits than GaussianImage. We attribute this to Gaussifier's density-adaptive EMV placement, which GaussianImage's optimization from random positions does not recover.