acceptodds
Under review as a conference paper at ICLR 2027

From Pixels to Compact Gaussians: Generalizable 2D Gaussian Image Representation via Error-Aware Graph Coarsening

Abstract

Existing 2D Gaussian image representations often learn to place primitives from scratch or optimize them separately for each image. We observe that an image itself already provides an analytic starting point: each pixel supplies a Gaussian center and color, while its local structure tensor determines an anisotropic covariance. Rasterizing this dense field achieves approximately – dB PSNR without training or per-image optimization. This high-fidelity but overcomplete initialization reframes compact Gaussian representation as a budget-constrained simplification problem. This overcomplete foundation shifts the core objective from synthesizing primitives to strategically pruning them under reconstruction feedback. To this end, we introduce Error-aware Pixel-Inductive Gaussians (EPIG), a bottom-up framework that alternates error-aware graph coarsening with reconstruction-driven refinement. A merging network selects neighboring primitives for moment-matched contraction, and a refinement network updates the survivors using the rendering residual. Repeating these steps yields a compact representation in a few feed-forward stages without test-time optimization. Across three budgets on DIV2K and zero-shot Kodak, EPIG improves PSNR over the evaluated feed-forward baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
2D Gaussian splatting, error-aware, feed-forward, graph coarsening

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_HBaDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Error-aware Pixel-Inductive Gaussians (EPIG) for representing an image with a prescribed number of 2D Gaussian primitives. It starts from an analytic, pixel-wise Gaussian field whose covariances are derived from local structure tensors. A learned graph coarsening module selects neighboring primitives for moment-matched contraction, while a refinement network uses rendering residuals to update the surviving Gaussian attributes. The refined state then informs subsequent merge decisions. On DIV2K and zero-shot Kodak, EPIG achieves higher PSNR and MS-SSIM than the evaluated feed-forward baselines at all three tested budgets, at the cost of longer inference time.

Critical strengths
  1. Consistent reconstruction gains over the evaluated feed-forward baselines. EPIG achieves consistent improvements in PSNR and MS-SSIM across the tested Gaussian budgets. On DIV2K, its PSNR exceeds that of AIR by 1.35, 2.74, and 2.28 dB at 10k, 20k, and 50k primitives, respectively. The improvements also transfer to Kodak without additional training or image-specific optimization, providing evidence that the learned pipeline generalizes beyond the training dataset.
Critical weaknesses
  1. The specific benefit of the dense-to-sparse formulation is not sufficiently established. The paper presents dense analytic initialization followed by progressive coarsening as its central methodological contribution. However, both sparse-to-dense and dense-to-sparse approaches ultimately seek to represent the same image with a prescribed number of Gaussian primitives. The direction of construction changes, but the final representation and its objective remain essentially the same. The paper does not clearly explain which limitation of sparse-to-dense construction is specifically addressed by starting from one Gaussian per pixel and progressively merging them. Although the reported results demonstrate the effectiveness of the complete system, they do not establish whether its advantage comes from this change in formulation or from particular design choices within the pipeline. The distinctive value of the proposed formulation therefore remains insufficiently demonstrated.

  2. The ablations do not adequately distinguish the contributions of the individual components. Table 2 suggests that refinement plays a particularly important role in the reported performance. At a budget of 50k primitives, learned merging without refinement improves PSNR from 24.99 to 25.92 dB over handcrafted merging, whereas enabling refinement raises it to 31.73 dB. This pattern warrants a closer examination of where the gains originate. The merging-only variant disables refinement after joint training, and refinement also changes the states on which subsequent merge decisions are based. Consequently, the current comparison does not clearly establish how much the learned merging policy contributes when refinement is present, or whether simpler merging strategies could achieve similar results with comparable refinement. Runtime is also missing for the ablation variants, making it difficult to assess the computational cost associated with each improvement.

Questions for authors
  1. What specific advantage does dense-to-sparse construction offer over sparse-to-dense construction? Please clarify which limitation of existing sparse-to-dense approaches motivates reversing the construction process, and why starting from a pixel-wise Gaussian field should address that limitation. Since both approaches ultimately produce a compact Gaussian representation, the central question is what is gained by this change in formulation.

  2. Can more detailed ablations clarify the roles of merging and refinement? Given the substantial improvement when refinement is enabled, it would be useful to compare handcrafted and learned merging with refinement modules of comparable capacity, trained consistently for each variant. This would help determine how much the learned merging policy contributes to the final reconstruction quality. An analysis of the number of refinement steps would further clarify the role of repeated corrections. Please report both reconstruction quality and per-image encoding time at each Gaussian budget for these variants, so that the benefits and costs of the components can be assessed together.

2 comments

We sincerely thank Reviewer HBaD for recognizing our consistent reconstruction gains across budgets and zero-shot generalization to Kodak. Before presenting our new decoupled ablations, we would like to clarify the core theoretical motivation behind EPIG (addressing W1 and Q1).

1. Paradigm Shift from Primitive Synthesis to Analytically Anchored Reduction (W1, Q1)

We respectfully clarify that EPIG is not motivated by merely reversing the construction direction from sparse-to-dense to dense-to-sparse. Rather, our motivation is to exploit the explicit geometric prior already present in an image, establishing a principled, analytically anchored starting point for constructing compact Gaussian representations.

(a). The Fundamental Bottleneck of Top-Down Primitive Synthesis: Existing feed-forward 2D Gaussian methods (e.g., AIR, Fast-2DGS) follow a primitive synthesis paradigm: starting from coarse grids, random points, or weakly grounded tokens, they force predictors to simultaneously solve two coupled problems from scratch—(i) where primitives belong on the 2D plane, and (ii) how their anisotropic shapes are parameterized. Under tight budgets (N=10k–20kN=10\text{k}\text{--}20\text{k}), inaccurate allocation misses thin structures and sharp boundaries, causing blurring or streaking artifacts (Fig. 3).

(b). From Primitive Synthesis to Image-Induced Geometric Priors: We revisit this problem from a different perspective: Does an image already contain sufficient geometric information to construct meaningful Gaussian primitives without learning their initial placement from scratch?

Our insight is that every pixel pp already induces a geometrically grounded 2D Gaussian analytically: its coordinate and RGB supply μ(p)=p\boldsymbol{\mu}(p)=p, c(p)=I(p)\boldsymbol{c}(p)=I(p), α(p)=1\alpha(p)=1, while its local structure tensor J(p)=k∗(∇I(p)∇I(p)T)J(p)=k\ast(\nabla I(p)\nabla I(p)^{T}) summarizes gradient variation via eigenvalues ν1(p)≥ν2(p)\nu_{1}(p)\geq\nu_{2}(p) and eigenvector matrix R(θ(p))R(\theta(p)) (aligned with local edge normal and tangent). With anisotropy ρ(p)=ν1(p)/ν2(p)\rho(p)=\sqrt{\nu_{1}(p)/\nu_{2}(p)} (bounded in flat regions), we construct the covariance as:

Σ(p)=R(θ(p))diag(ρ(p)−1,ρ(p))R(θ(p))T.\Sigma(p) = R(\theta(p))\mathrm{diag}(\rho(p)^{-1}, \rho(p))R(\theta(p))^{T}.

Rasterizing this training-free dense field already achieves ~25–30 dB PSNR (e.g., 25.784 dB in Fig. 2 without learning or per-image optimization—surpassing 1,000-step 3DGS at 20.62 dB and Fast-2DGS at 15.17 dB at 10k primitives). Because locations and orientations are analytically anchored to image structures from the start, EPIG eliminates blind primitive placement and reframes compact representation as budget-constrained, structure-preserving graph reduction—guaranteeing exact budget compliance (∣G(T)∣=N|G^{(T)}|=N) with one model.

(c). Learned Error-Aware Reduction Beyond Heuristic Merging: Crucially, EPIG is not a static heuristic merging rule. While local handcrafted merging relies on myopic pairwise rules and degrades thin structures (Fig. 4, 24.99 dB at 50k), EPIG is an end-to-end feed-forward learned reduction framework alternating a Merging Prediction Network C(θ)\mathcal{C}(\theta) (deciding where capacity can be removed) with a Reconstruction-driven Refinement Network R(ϕ)\mathcal{R}(\phi) (deciding how remaining capacity is redistributed via rendering residuals I~−I\widetilde{I}-I). Joint closed-loop training teaches C(θ)\mathcal{C}(\theta) to preserve topological structures maximizing post-refinement fidelity (+6.74 dB over handcrafted merging at 50k).

(d). Direct Controlled Validation Against Top-Down Initializations: To prove empirically that this advantage stems from analytically anchored reduction rather than merely from R(ϕ)\mathcal{R}(\phi), we applied our same frozen R(ϕ)\mathcal{R}(\phi) to three top-down NN-Gaussian initializations—Uniform Grid, Random Pixel Sampling, and SLIC Superpixels (initialized with region moments)—on DIV2K (100 validation images):

Budget NNGrid+R(ϕ)\mathcal{R}(\phi)Rand+R(ϕ)\mathcal{R}(\phi)SLIC (0-step)SLIC+R(ϕ)\mathcal{R}(\phi) (Matched)SLIC+R(ϕ)\mathcal{R}(\phi) (Oracle 1–8)EPIG (Full)Gain vs. SLIC (Matched / Oracle)
10k21.8420.1523.7323.0925.7926.73+3.64 / +0.94 dB
20k24.3122.4825.4127.4928.0629.02+1.53 / +0.96 dB
40k26.5224.8926.4829.4829.5830.91+1.43 / +1.33 dB
50k27.2825.6426.7530.4330.5431.73+1.30 / +1.19 dB
100k28.9027.1227.0830.0130.8633.11+3.10 / +2.25 dB

Even when granting SLIC an Oracle Best-Step advantage (picking each image's best step across 1–8 steps), EPIG still leads by +0.94 to +2.25 dB (winning on 94–98 of 100 images).

2. Decoupling Merging vs. Refinement, Refinement Steps, and Variant Runtimes (W2, Q2)

(a) Why Learned Merging Is Essential When Refinement Is Present (Post-Refinement Reachability) Comparing SLIC (a strong top-down region-merging proxy) against EPIG before and after applying the exact same refinement module R(ϕ)\mathcal{R}(\phi) reveals the exact mechanism of joint training:

  • Before refinement (0-step): SLIC's raw rendering (23.73 / 25.41 / 26.75 dB at 10k / 20k / 50k) is actually higher than EPIG Merge-only (23.24 / 24.48 / 25.92 dB).
  • After identical refinement: EPIG surges ahead of SLIC+R(ϕ)\mathcal{R}(\phi) by +3.64 dB (10k), +1.53 dB (20k), and +1.30 dB (50k) (and similarly beats Grid+R(ϕ)\mathcal{R}(\phi) by 4.4–4.9 dB).
  • Interpretation: Because C(θ)\mathcal{C}(\theta) and R(ϕ)\mathcal{R}(\phi) are trained end-to-end in a closed loop, C(θ)\mathcal{C}(\theta) is not optimized to maximize unrefined moment-matching PSNR (which static rules or SLIC already optimize locally). Instead, C(θ)\mathcal{C}(\theta) learns to preserve thin topological structures and spatial degrees of freedom that R(ϕ)\mathcal{R}(\phi) can sharpen, optimizing for post-refinement reconstruction fidelity. Furthermore, masking the rendering residual stream in R(ϕ)\mathcal{R}(\phi) and re-adapting for 20k training steps still degrades PSNR by −1.44-1.44 to −2.07-2.07 dB across 10k–50k budgets, confirming that closed-loop rendering feedback is critical.

(b) Effect of the Number of Refinement Steps In EPIG's default rollout, we execute 1 refinement step per coarsening stage once n(t)≤150kn^{(t)} \le 150\text{k} (4–8 stages total depending on image resolution and budget NN). To analyze repeated corrections, we evaluated applying T=1…10T=1\dots 10 additional refinement steps at the target budget NN:

Budget NNDefault Rollout (1 step/stage)+1 Extra Step (T=1T=1)+10 Extra Steps (T=10T=10)Default Rollout Time (s)
10k26.73 dB26.94 dB (+0.21, 82% imgs)18.24 dB2.70 s
20k29.02 dB29.39 dB (+0.37, 95% imgs)21.46 dB2.40 s
50k31.73 dB32.19 dB (+0.46, 95% imgs)25.76 dB1.92 s

R(ϕ)\mathcal{R}(\phi) acts as an amortized single-step corrector tailored to each contraction stage: 1 extra step yields a modest +0.21∼+0.46+0.21\sim+0.46 dB gain (at 0.13–0.45 s/step), whereas looping R(ϕ)\mathcal{R}(\phi) for T=10T=10 steps pushes Gaussians off the training manifold. Note also why smaller budgets take slightly longer in default rollout (2.70 s at 10k vs. 1.92 s at 50k): reaching N=10kN=10\text{k} requires more coarsening-and-refinement rounds below the 150k threshold than stopping at N=50kN=50\text{k}.

(c) Per-Image Encoding Time Breakdown Across Variants On DIV2K (RTX 3090), we report the runtime and quality of each ablation variant across budgets:

Variant10k PSNR / Time20k PSNR / Time50k PSNR / Time
Handcrafted merging only23.42 dB / 0.42 s24.42 dB / 0.39 s24.99 dB / 0.34 s
Learned merging only (w/o R(ϕ)\mathcal{R}(\phi))23.24 dB / 0.58 s24.48 dB / 0.54 s25.92 dB / 0.47 s
EPIG Full (Learned merge + R(ϕ)\mathcal{R}(\phi))26.73 dB / 2.70 s29.02 dB / 2.40 s31.73 dB / 1.92 s

While merging only runs in 0.34–0.58 s, adding the closed-loop refinement stages (~1.5–2.1 s total across rounds) yields a massive +3.49 to +5.81 dB gain while still eliminating all per-image gradient optimization.

We will include all these new analyses, tables, and runtime breakdowns in the revised paper, and hope they address your concerns.

Reviewer_LDSaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EPIG, a bottom-up framework for constructing compact 2D Gaussian image representations through iterative merging and refinement. It first initializes a dense set of 2D Gaussians, with one Gaussian assigned to each pixel. The model then predicts merging scores for Gaussian pairs on a region-adjacency graph and contracts high-scoring pairs to progressively reduce the Gaussian count. Subsequently, the remaining Gaussians are refined using a reconstruction-aware refinement network with KNN-based attention. Experimental results show that EPIG achieves higher reconstruction quality than both optimization-based methods and top-down feed-forward methods under the same Gaussian budgets (10K, 20K, and 50K).

Critical strengths
  • The paper reformulates 2D Gaussian image reconstruction as a bottom-up process, which differs from existing top-down methods. The proposed analytic initialization and learned simplification process convert dense pixel-wise Gaussians into a compact representation at a user-specified budget, without any per-image optimization. This framework design clearly differs from previous methods.
  • In addition, the reconstruction-aware refinement network improves the fidelity of the compact representation in a feed-forward manner. In particular, updating the Gaussian parameters based on the reconstruction error leads to a substantial gain in fidelity.
Critical weaknesses
  • The paper proposes a bottom-up framework, but it does not demonstrate that this formulation is actually more effective than top-down approaches. In Table 2, the variants without refinement perform worse than the existing feed-forward methods. This indicates that the reconstruction quality gain mainly comes from the refinement network rather than from the bottom-up design. A comparison between bottom-up and top-down designs is necessary.
  • While the comparison with existing methods is sufficient, the ablation of the proposed pipeline is insufficient. The design choices, such as the analytic pixel-to-Gaussian initialization, the handcrafted merging prior, the edge scores, and the inputs of the refinement network, are not validated. For the completeness of the paper, an ablation study on each component should be provided.
  • The bottom-up approach may be less efficient for high-resolution images. In top-down approaches, the number of Gaussians gradually increases, so the final set serves as the memory upper bound. In contrast, the bottom-up approach starts from a dense set, which requires a large amount of memory and computation at the early stages. In particular, operations such as KNN attention become a significant burden as the number of Gaussians increases.
Questions for authors
  • Experiments and validation showing that the bottom-up approach is better than the top-down approach are needed.
  • An ablation study on each component should be provided.
  • The bottom-up approach seems to require a large amount of memory. How does the memory usage of each method compare?
  • With sufficient optimization steps, per-image optimization achieves better results. If the refinement is also applied repeatedly, like per-image optimization, which approach yields better results?
  • A comparison between refinement and per-image optimization is also needed. Performing per-image optimization instead of refinement might be a higher-quality and more efficient approach.
  • Quickly generating initial Gaussians with per-image optimization and attaching only the refinement module could also lead to good results. A discussion of the performance difference between this and the bottom-up merging approach would be helpful.
3 comments

We sincerely thank Reviewer LDSa for the encouraging feedback, particularly for recognizing the novelty of our bottom-up reformulation and exact budget control without per-image optimization. Below, we address all weaknesses and questions with controlled experiments on the 100-image DIV2K validation set (2×2\times bicubic downsampled).

1. Bottom-Up Coarsening vs. Top-Down & Fast Initializations (W1, Q1, Q6)

To isolate the contribution of our bottom-up coarsening from the refinement network R(ϕ)\mathcal{R}(\phi), we constructed three top-down / fast heuristic baselines that initialize exact NN Gaussians via—Uniform Grid, Random Pixel Sampling, and structure-adaptive SLIC Superpixels (initialized with region color and covariance moments)—and fed them directly into our same frozen refinement network R(ϕ)\mathcal{R}(\phi). We report matched refinement budgets and an Oracle Best-Step (picking each image's highest-PSNR step across 1–8 steps):

Budget NNGrid+R(ϕ)\mathcal{R}(\phi)Rand+R(ϕ)\mathcal{R}(\phi)SLIC (0-step)EPIG Merge-onlySLIC+R(ϕ)\mathcal{R}(\phi) (Matched)SLIC+R(ϕ)\mathcal{R}(\phi) (Oracle 1–8)EPIG (Full)Gain vs. SLIC (Matched / Oracle)
10k21.8420.1523.7323.2423.0925.7926.73+3.64 / +0.94 dB
20k24.3122.4825.4124.4827.4928.0629.02+1.53 / +0.96 dB
40k26.5224.8926.4825.5629.4829.5830.91+1.43 / +1.33 dB
50k27.2825.6426.7525.9230.4330.5431.73+1.30 / +1.19 dB
100k28.9027.1227.0826.7930.0130.8633.11+3.10 / +2.25 dB
  • Substantial Gains Over Top-Down Initializations: Under matched refinement budgets, EPIG beats Grid+R(ϕ)\mathcal{R}(\phi) and Random+R(ϕ)\mathcal{R}(\phi) by 3.8–6.5 dB, and surpasses SLIC+R(ϕ)\mathcal{R}(\phi) by +3.64 dB (10k), +1.53 dB (20k), +1.43 dB (40k), +1.30 dB (50k), and +3.10 dB (100k). Even against Oracle Best-Step, EPIG still leads by +0.94 to +2.25 dB (winning on 94–98 of 100 images).
  • Why Merge-only Appears Modest in Table 2: Before refinement, SLIC (0-step: 23.73–27.08 dB) exceeds EPIG Merge-only (23.24–26.79 dB). Yet once the exact same R(ϕ)\mathcal{R}(\phi) is applied, EPIG surges ahead by +1.30 to +3.64 dB. Joint closed-loop training teaches C(θ)\mathcal{C}(\theta) not to maximize unrefined moment-matching PSNR, but to preserve thin topological structures that maximize post-refinement fidelity.

2. Comprehensive Ablation Studies on Each Component (W2, Q2)

Beyond ablating the analytic initialization above, we ablated the remaining components:

  • Refinement Network Input Stream: Masking the rendering residual stream (I~−I\widetilde{I}-I and B(I~)−B(I)B(\widetilde{I})-B(I) in Eq. 7) while retaining Gaussian attributes and image-conditioned priors—even after 20,000 re-adaptation steps—degrades PSNR across all budgets: 10k: 26.73→25.2926.73 \to 25.29 (−1.44-1.44 dB, LPIPS +0.069+0.069); 20k: 29.02→26.9529.02 \to 26.95 (−2.07-2.07 dB, LPIPS +0.073+0.073); 40k: 30.91→28.8930.91 \to 28.89 (−2.02-2.02 dB, LPIPS +0.050+0.050); 50k: 31.73→29.8231.73 \to 29.82 (−1.91-1.91 dB, LPIPS +0.044+0.044); 100k: 33.11→32.0033.11 \to 32.00 (−1.11-1.11 dB, LPIPS +0.023+0.023). Explicit rendering discrepancy provides indispensable error signals.
  • Handcrafted Merging Cost Components (C(a,b)C(a,b) in Eq. 4): Removing the Mahalanobis barrier causes the largest drop (−2.83-2.83 dB at 10k, −1.73-1.73 dB at 20k) by allowing elongated merges across object boundaries; removing the color term drops PSNR by −1.43-1.43 dB (10k) and −1.18-1.18 dB (20k); the gradient energy term acts as a spatial allocation regularizer (±0.1\pm 0.1 dB on raw merging PSNR).
  • Merging Predictor Capacity (C(θ)\mathcal{C}(\theta)): Operating on up to HW≈400kHW \approx 400\text{k} nodes early on, keeping C(θ)\mathcal{C}(\theta) lightweight (0.17M parameters, 262.3 GFLOPs in stage 1) is crucial for low latency; wider/deeper variants yielded <0.05<0.05 dB gain while increasing runtime.

3. GPU Memory Usage & High-Resolution Efficiency (W3, Q3)

We appreciate this important question regarding dense-stage scalability. As described in Appendix A.4, EPIG completely bypasses the refinement network and KNN attention whenever the primitive count exceeds 150k (n(t)>150kn^{(t)} > 150\text{k}). Consequently:

  1. During early dense stages (HW→150kHW \to 150\text{k}), EPIG only runs the ultra-lightweight 0.17M merging predictor over local 8-neighbor region-adjacency edges (O(n(t))O(n^{(t)}) linear memory and compute) followed by analytic moment matching.
  2. Spatial KNN attention (k=16k=16) is only invoked after the graph has already been coarsened below 150k primitives.

We benchmarked peak inference GPU memory (MB, averaged over DIV2K validation images) across 10 methods at both 10k and 50k budgets:

Method10k Memory (MB)50k Memory (MB)Method10k Memory (MB)50k Memory (MB)
EPIG (Rollout)10,35110,355Image-GS255261
EPIG (Hybrid)10,35310,355Instant-GI253253
AIR2,9932,993LIG211215
Fast-2DGS8538593DGS / GaussianImage88 / 71144 / 85

Notice that EPIG's peak memory is virtually constant across budgets (10,351 MB at 10k vs. 10,355 MB at 50k) and easily fits on a single consumer GPU. This footprint is dominated by activation caches of the shared frozen feature encoder (DINOv2 ViT-B/14 + ResNet-18) during image feature extraction rather than dense-stage Gaussian graphs.


4. Repeated Refinement vs. Per-Image Gradient Optimization & Warm-Start (Q4, Q5)

To clarify the relationship between feed-forward refinement and per-image gradient optimization, we conducted two complementary experiments on DIV2K:

(a) Applying R(ϕ)\mathcal{R}(\phi) Repeatedly vs. Per-Image Gradient Fine-Tuning (Q4 & Q5) Starting from EPIG's feed-forward rollout output, we compared (i) repeatedly applying the trained refinement head R(ϕ)\mathcal{R}(\phi) for T=1…10T=1\dots 10 extra steps without further merging, versus (ii) running lightweight per-image gradient fine-tuning (Adan optimizer) on the surviving Gaussians:

Budget NNEPIG RolloutRepeated R(ϕ)\mathcal{R}(\phi) (Best Step)Repeated R(ϕ)\mathcal{R}(\phi) (T=10T=10)Per-Image Grad Fine-Tune (Best)
10k26.73 dB26.94 dB (+0.21, step 1)18.24 dB29.10 dB (+2.37, ~103 steps)
20k29.02 dB29.39 dB (+0.37, step 1)21.46 dB32.11 dB (+3.09, ~159 steps)
40k30.91 dB31.55 dB (+0.64, step 1)24.74 dB35.31 dB (+4.40, ~303 steps)
50k31.73 dB32.19 dB (+0.46, step 1)25.76 dB36.45 dB (+4.72, ~371 steps)
  • Why R(ϕ)\mathcal{R}(\phi) Should Not Be Looped Independently: R(ϕ)\mathcal{R}(\phi) is trained as an amortized single-step corrector interleaved with graph contraction. Applying 1 extra post-rollout refinement step still yields a consistent gain of +0.21 to +0.64 dB (improving 82–97% of images), but repeating R(ϕ)\mathcal{R}(\phi) for T=10T=10 steps pushes Gaussian states outside the training distribution (−6.0-6.0 to −8.5-8.5 dB drift).
  • Efficiency Complementarity (Q5): Per-step, gradient fine-tuning costs only 4.6–7.3 ms/step—which is about 100$\times$ faster per step than a forward pass of R(ϕ)\mathcal{R}(\phi) (0.13–0.45 s/step)—and boosts PSNR by +2.37 to +4.72 dB in just ~100–370 steps (0.5–2.5 s). Thus, EPIG excels in zero-optimization feed-forward deployment, while brief gradient fine-tuning on EPIG's output offers the best efficiency when test-time optimization is permitted.

(b) EPIG as a Warm-Start for Standard Per-Image Optimization (Q5) When using EPIG's output to initialize standard GaussianImage optimization (Adan, lr=10−310^{-3}) compared to standard random initialization under identical optimizer settings:

  • Time-to-Quality Speedup: EPIG accelerates convergence to target PSNR thresholds by 2.0–12.2$\times$ at 10k (e.g., 12.2×12.2\times faster to reach 20 dB, 2.0×2.0\times faster to reach 34 dB) and 1.8–4.4$\times$ at 20k.
  • Final Quality Gain: Even after 5,000 optimization steps, EPIG-initialized optimization still outperforms random initialization by +1.81 dB at 10k (98% image win rate) and +1.07 dB at 20k (94% image win rate).

We will incorporate all new experimental tables and analyses into the revised manuscript. We hope these results address your concerns and kindly ask you to reconsider your rating.

Thank you for the detailed responses and the additional experiments. My concern on the ablation study (W2, Q2) is resolved. I also appreciate the clarification on iterative refinement (Q4). However, the following concerns remain.

1. Bottom-up vs. top-down (W1, Q1, Q6).

The comparison with a frozen refinement network is unfair. The network is trained on the outputs of the EPIG merging predictor, so lower performance on out-of-distribution inputs is expected. In addition, the comparison should involve effective top-down methods such as AIR and Fast-2DGS, or initializations obtained by short per-image optimization, rather than simple heuristics.

2. GPU memory (W3, Q3).

I understand that the peak memory is nearly constant across budgets. However, it is much larger than that of existing feed-forward methods, about 3.5 times that of AIR and 12 times that of Fast-2DGS. These numbers are measured on 2x downsampled images. At the original resolution, the memory would likely increase by about four times and exceed the capacity of a single consumer GPU.

3. Refinement vs. per-image optimization (Q5).

The two are still not compared directly. The authors only report per-image optimization applied on top of the refined output, which does not show whether refinement offers an advantage over optimization, such as a faster PSNR improvement under the same time budget.

Despite the rebuttal, the authors have not clearly demonstrated the advantage of the bottom-up approach over effective top-down methods, the GPU memory overhead remains a significant concern, and the benefit of the refinement network over per-image optimization is still not demonstrated. Therefore, I will maintain my original rating.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Core Motivation and Analytically Anchored Reduction

We sincerely thank the Area Chair and all reviewers for your time, thoughtful evaluations, and constructive feedback. In this overall response, we first clarify the core motivation and coupled mechanism underlying EPIG, and then address the shared questions raised across the reviews with new controlled experiments.

1. Paradigm Shift from Primitive Synthesis to Analytically Anchored Reduction

We would like to clarify that EPIG is not motivated by simply reversing the construction direction from sparse-to-dense to dense-to-sparse. Rather, our motivation is to exploit the geometric prior already present in an image, which provides a principled starting point for constructing Gaussian representations. This observation fundamentally changes how we approach Gaussian representation construction.

From learning primitive synthesis to exploiting image-induced geometric priors

Existing feed-forward 2D Gaussian methods typically follow a top-down primitive synthesis paradigm: they directly predict a compact set of Gaussian primitives from image features or progressively increase primitive capacity from coarse or weakly grounded initializations. Although their implementations differ, they generally need to learn where primitives should be placed and how their anisotropic shapes are parameterized. These two problems are tightly coupled, particularly under a limited primitive budget. An inaccurate initial allocation can under-represent thin structures, high-frequency boundaries, and other spatially localized details.

We revisit this problem from a different perspective: Does an image already contain sufficient geometric information to construct meaningful Gaussian primitives without learning their initial placement from scratch?

Our key observation is that every pixel pp can be analytically lifted into a 2D Gaussian using its existing image attributes. Specifically, the pixel coordinate and RGB value supply μ(p)=p\boldsymbol{\mu}(p)=p, c(p)=I(p)\boldsymbol{c}(p)=I(p), and α(p)=1\alpha(p)=1, while the local structure tensor J(p)=k∗(∇I(p)∇I(p)T)J(p)=k\ast(\nabla I(p)\nabla I(p)^{T}) summarizes directional gradient variation via its eigenvalues ν1(p)≥ν2(p)\nu_{1}(p)\geq\nu_{2}(p) and eigenvector matrix R(θ(p))R(\theta(p)) (aligned with the local edge normal and tangent). Defining the anisotropy ratio ρ(p)=ν1(p)/ν2(p)\rho(p)=\sqrt{\nu_{1}(p)/\nu_{2}(p)} (bounded in degenerate flat regions), we analytically construct the covariance and additive spatial support mass as:

Σ(p)=R(θ(p))diag(ρ(p)−1,ρ(p))R(θ(p))T,w(p)=det⁡Σ(p).\Sigma(p) = R(\theta(p))\mathrm{diag}(\rho(p)^{-1}, \rho(p))R(\theta(p))^{T}, \qquad w(p) = \sqrt{\det\Sigma(p)}.

Thus, each Gaussian is compressed across the local edge and elongated along its tangent, while approximately isotropic primitives are induced in flat regions. Consequently, the image itself analytically induces an overcomplete Gaussian field G(0)=A(I)\mathcal{G}^{(0)}=\mathcal{A}(I) of size HWHW.

This lifting is entirely training-free and requires no per-image optimization. Importantly, direct rasterization of this dense field already achieves approximately 25–30 dB PSNR. For example, the first result in the second row of Fig. 2 achieves 25.784 dB with 393,216 Gaussians, without any learning or test-time optimization—already surpassing 1,000-step 3DGS (20.62 dB), GaussianImage (13.05 dB), and Fast-2DGS (15.17 dB) at 10k primitives.

This result should not be interpreted as a weak compact-representation baseline. Instead, it demonstrates that the image-induced Gaussian field is already a meaningful, geometrically grounded representation before any learning-based processing takes place. Its initial primitive locations and anisotropic orientations are analytically anchored to pixel coordinates and local image structures, rather than being blindly predicted from scratch.

The analytically induced Gaussian field therefore provides a geometrically grounded starting point. The remaining challenge is how to reduce this overcomplete representation to a prescribed primitive budget while preserving reconstruction fidelity.

Crucially, rather than relying on static heuristic merging rules, EPIG solves this as an end-to-end feed-forward learning framework (with zero test-time optimization) that jointly trains a learned merging predictor C(θ)\mathcal{C}(\theta) and a reconstruction-driven refinement network R(ϕ)\mathcal{R}(\phi), as detailed below.

AuthorsOpenReview ↗

Why Error-aware Merging and Reconstruction-driven Refinement Must Be Coupled

2. Why Error-aware Merging and Reconstruction-driven Refinement Must Be Coupled

Starting from the analytically induced Gaussian field, the central challenge is to reduce the number of primitives while preserving reconstruction fidelity. A straightforward solution would be to apply handcrafted graph merging followed by per-image optimization. However, handcrafted merging relies on local heuristics and cannot directly optimize end-to-end rendering quality, while per-image optimization requires iterative gradient updates for every test image.

EPIG addresses these limitations by jointly learning which Gaussians to merge and how to refine the surviving primitives through explicit rendering feedback.

A. Learning where representation capacity can be removed

We first organize the current Gaussian field as a dynamic region-adjacency graph. Each node represents a Gaussian together with its associated image region, and an edge connects two nodes whose regions share a boundary. This ensures that merging is spatially structured rather than performed between arbitrary primitives.

A handcrafted merging cost based on color discrepancy, gradient content, and geometric separation provides an initial structural prior. However, such a local heuristic cannot directly determine which contractions will lead to the best final rendering quality. We therefore introduce a Merging Prediction Network C(θ)\mathcal{C}(\theta) (0.17M parameters) that extracts node and edge descriptors and predicts the compatibility of neighboring Gaussian pairs, conditioned on the current primitive budget log⁡(n(t)/N)\log(n^{(t)}/N). We select mutual-best edges to obtain non-overlapping pairs, capped by the remaining reduction required to hit the exact target budget NN, and replace each pair with its mass-weighted moment-matched Gaussian (xabx^{ab} and Σab\Sigma^{ab}). Nevertheless, moment preservation does not guarantee rendering fidelity: merging two locally compatible Gaussians can still introduce image-space and perceptual reconstruction errors.

B. Learning how to redistribute the remaining representation capacity

To compensate for the errors introduced by contraction, EPIG introduces a Reconstruction-driven Refinement Network R(ϕ)\mathcal{R}(\phi). After merging, we render the contracted Gaussian set to obtain I~=S(G~(t+1))\widetilde{I}=\mathcal{S}(\widetilde{\mathcal{G}}^{(t+1)}) and explicitly measure its discrepancy from the target image II. A shared frozen encoder extracts multi-layer features from both images, and we combine the feature-space discrepancy and pixel-space residual via Gaussian-weighted pooling:

r(i)=P(i)[ϕf(B(I~)−B(I))+ϕp(I~−I)].r^{(i)} = \mathcal{P}^{(i)}\left[\phi_{f}(B(\widetilde{I})-B(I)) + \phi_{p}(\widetilde{I}-I)\right].

This localized reconstruction-error descriptor is combined with the contracted Gaussian attributes, recurrent state, and remaining budget information. A spatial KNN-attention Graph Transformer then exchanges information among neighboring Gaussians and predicts attribute updates (position, log-eigenvalues, orientation, color, and opacity). Importantly, refinement does not introduce new primitives or change pixel-region assignments; it adjusts the surviving primitives so that the remaining representation capacity is used more effectively.

C. Why the two components must be jointly trained in a closed loop

Merging and refinement solve complementary but interdependent problems:

  • Merging determines where representation capacity can be removed.
  • Refinement determines how the remaining capacity should be redistributed.

Crucially, the rendering residual explicitly guides refinement, while the refined Gaussian state is passed to the next merging stage. Through joint training with a straight-through Gumbel surrogate, reconstruction losses computed after refinement directly supervise the merging decisions. Consequently, the merging policy is optimized according to the reconstruction quality achievable after refinement, rather than merely the immediate quality after contraction:

G(t)⟶C(θ)G~(t+1)⟶R(ϕ)G(t+1).\mathcal{G}^{(t)} \overset{\mathcal{C}(\theta)}{\longrightarrow} \widetilde{\mathcal{G}}^{(t+1)} \overset{\mathcal{R}(\phi)}{\longrightarrow} \mathcal{G}^{(t+1)}.

The refined state carries reconstruction feedback into the next coarsening stage, creating a closed-loop reduction process in which merging and refinement are jointly optimized rather than treated as independent post-processing modules.

AuthorsOpenReview ↗

Controlled Experiments, Complexity, and Baseline Comparisons

3. Additional Experiments Validating the Coupled Mechanism & Practical Value

We evaluate our hypotheses on the 100-image DIV2K validation set (2×2\times bicubic downsampled) across five primitive budgets (N∈10k,20k,40k,50k,100kN \in \\{10\text{k}, 20\text{k}, 40\text{k}, 50\text{k}, 100\text{k}\\}).

A. Controlled top-down initialization comparison

To examine whether the improvement comes merely from applying a powerful refinement network, we compare EPIG against uniform-grid, random-pixel, and structure-adaptive SLIC superpixel initializations. Each top-down baseline starts with exactly NN Gaussians and is evaluated using the same frozen refinement head R(ϕ)\mathcal{R}(\phi) under both matched refinement budgets and an Oracle Best Step (picking each image's highest-PSNR step across 1–8 refinement steps):

Budget NNGrid+R(ϕ)\mathcal{R}(\phi)Rand+R(ϕ)\mathcal{R}(\phi)SLIC (0-step)Merge-onlySLIC+R(ϕ)\mathcal{R}(\phi) (Matched)SLIC+R(ϕ)\mathcal{R}(\phi) (Oracle 1–8)EPIG (Full)Gain vs. SLIC (Matched / Oracle)
10k21.8420.1523.7323.2423.0925.7926.73+3.64 / +0.94 dB
20k24.3122.4825.4124.4827.4928.0629.02+1.53 / +0.96 dB
40k26.5224.8926.4825.5629.4829.5830.91+1.43 / +1.33 dB
50k27.2825.6426.7525.9230.4330.5431.73+1.30 / +1.19 dB
100k28.9027.1227.0826.7930.0130.8633.11+3.10 / +2.25 dB

EPIG consistently outperforms SLIC by 1.30–3.64 dB under matched refinement budgets, and remains ahead by 0.94–2.25 dB even against Oracle Best Step (winning on 94–98 of 100 images). More importantly, the merge-only comparison reveals why joint merging and refinement matter: when refinement is disabled, the raw rendering quality of EPIG's merged outputs (23.24–26.79 dB) is lower than that of SLIC zero-refinement initialization (23.73–27.08 dB). Yet after applying the same refinement head, EPIG surpasses SLIC across all budgets. This indicates that through joint training, C(θ)\mathcal{C}(\theta) learns to preserve thin topological structures that yield superior post-refinement reconstruction quality.

B. Explicit rendering feedback is essential for refinement

Removing the rendering-residual input stream (I~−I\widetilde{I}-I and B(I~)−B(I)B(\widetilde{I})-B(I)) while retaining Gaussian-state and image-conditioned priors—even after re-adapting the model for 20k training steps—consistently degrades PSNR across all budgets by −1.44-1.44 dB (10k: 26.73→25.2926.73 \to 25.29), −2.07-2.07 dB (20k: 29.02→26.9529.02 \to 26.95), −2.02-2.02 dB (40k: 30.91→28.8930.91 \to 28.89), −1.91-1.91 dB (50k: 31.73→29.8231.73 \to 29.82), and −1.11-1.11 dB (100k: 33.11→32.0033.11 \to 32.00) (with LPIPS worsening by +0.023+0.023 to +0.073+0.073).

C. Refinement is an amortized correction operator, not an iterative optimizer

Across 10k–50k budgets, 1 extra post-rollout refinement step yields +0.21∼+0.56+0.21\sim+0.56 dB (on 82–97% of images), whereas repeating R(ϕ)\mathcal{R}(\phi) for T=10T=10 steps drives states off-manifold (−6.0-6.0 to −8.5-8.5 dB). In contrast, lightweight per-image gradient optimization on EPIG's output costs only 4.6–7.3 ms/step (100$\times$ faster per step than R(ϕ)\mathcal{R}(\phi)) and further boosts PSNR by +2.37 to +7.24 dB in ~103–753 steps.

D. Complexity, Trainable Parameters, Instant-GI, and Warm-Start

  • Parameters & Constant Memory: Of EPIG's 109.4M parameters, 87.3M are frozen ResNet18+DINOv2; EPIG has only 22.13M trainable parameters (0.17M C(θ)\mathcal{C}(\theta) + 21.96M R(ϕ)\mathcal{R}(\phi))—17.7$\times$ smaller than AIR (392.7M) and 4.5$\times$ smaller than Instant-GI (100.0M). Because refinement is bypassed when n(t)>150kn^{(t)} > 150\text{k}, peak GPU memory is constant across budgets (~10.35 GB at 10k and 50k).
  • Instant-GI & Warm-Start: Instant-GI (ICCV 2025, k=3k=3) averages N=76.9kN=76.9\text{k} (min 37.9k) and cannot reach 10k/20k budgets; at zero-optimization inference, EPIG outperforms it by +2.0 to +3.1 dB on matched budgets (31.73 dB at 50k vs. Instant-GI's 29.58 dB at 76.9k). Furthermore, using EPIG as a warm-start for GaussianImage optimization accelerates time-to-quality by 2.0–12.2$\times$ (10k) and 1.8–4.4$\times$ (20k).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.