acceptodds
Under review as a conference paper at ICLR 2027

Splatting Without Discrete Density Control: Lifted Gaussian-Mixture Kernel for CT Reconstruction

Abstract

3D Gaussian Splatting (3DGS) has become one of the key paradigms in 3D reconstruction, offering an efficient explicit representation method with real-time rendering capabilities. It has demonstrated significant potential in X-ray computed tomography (CT) reconstruction, but existing methods rely on a discrete Adaptive Density Control (ADC) mechanism based on hard-threshold cloning, splitting, and pruning decisions. The threshold parameters are coupled with hyperparameters such as the objective function design, leading to potential imbalances in the Gaussian distribution under different reconstruction conditions—particularly in ill-posed reconstruction—which require multiple manual adjustments and thus severely limit the performance ceiling of the reconstruction method. This paper utilizes a dimension-elevated Gaussian mixed-kernel affine transformation to convert the splitting and merging behaviors of Gaussian kernels into continuously differentiable motion trajectories, enabling their integration into the 3D reconstruction optimization chain without reliance on hyperparameter thresholds. Based on this mechanism, we have constructed a CT 3D reconstruction framework that allows users to allocate a budget for the number of Gaussian kernels to be reconstructed. Experiments show that, with the same number of kernels, our method outperforms the ADC-based 3DGS reconstruction method. Furthermore, when considering larger kernel budgets, ADC begins to show signs of fatigue or even a decline in performance without the use of any additional Gaussian kernel scheduling or pruning strategies, whereas our method maintains superior performance and continues to improve as the kernel budget increases.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computed Tomography, 3D Reconstruction, Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yoDmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Gaussian splatting method for CT reconstruction which works with fixed number of Gaussians, instead of using threshold-based density control. Main idea is to make each Gaussian as a slice of higher-dimensional Gaussian mixture, therefore splitting and merging can happen in continuous way during optimization. Authors also add a reallocation step under same budget and some adaptive rules for initialization and loss weights. Experiments show the method is better than R2-Gaussian at same kernel count, and the gap becomes larger when the budget increases.

Critical strengths
  • Exact control of the kernel count is useful in practice and it makes a controlled comparison across budgets possible. The budget sweep in Fig. 3 is one of the convincing part for me, since Lifted-GS is better than R2-Gaussian by about 0.7 to 1.9 dB at the largest budget on all four volumes while R2-Gaussian saturates.

  • The gain at the default matched count is small but consistent overall, including the real FIPS scans at 25 views. At ADC threshold 2e-4, R2-Gaussian almost does not densify (about 50.2k Gaussians), and Lifted-GS is still better by 0.37 to 0.68 dB while using about half of the initial points.

Critical weaknesses
  • The main claim is not really isolated by the experiments. Since x_ik and A_ik are free per component, the sliced field is the same function class as B plain 3D Gaussians, and c is redundant with the auxiliary means (shifting c and all b_ik together gives exactly the same field). So the benefit can only come from optimization dynamics, but this is not analyzed. Also the final method still relies on PAGR, which is a discrete hard selection with its own thresholds, and the comparison with R2-Gaussian changes initialization, loss, optimizer settings and fixed count at once. The only isolating evidence is one row of Table 3, on single volume and single seed with unspecified view count, so I am not convinced the lifting itself is the source of the improvement.

  • Several numbers in the tables do not match the appendix figures, and this matters because many margins in Table 1 are only 0.01 to 0.1 dB with one seed. Colon 75 and 180 views SSIM are swapped between two methods compared with Fig. 7 and Fig. 5/14, head-and-neck 25 views PSNR is 34.23/34.22 in Table 1 but 34.22/34.24 in Fig. 10, seashell 25 views SSIM of Lifted-GS is 0.9411 vs 0.9391 in Fig. 12, and seashell 75 views PSNR of R2-Gaussian is 43.99 vs 44.40. All of these differences go in favor of the proposed method. Table 2 also seems to pick the best points with different rules for two methods (R2-Gaussian SSIM 0.9507 and 0.9654, but 0.9517 and 0.9668 in Fig. 5/6). In current form, it is hard to trust the reported numbers.

Questions for authors
  • Please clarify which values are correct for the mismatches between Table 1/2 and Fig. 5, 6, 7, 10, 12, 14, 15, and which view count and kernel count are used in Table 3.

  • Could you report a plain fixed-count 3D Gaussian baseline with the same initialization, loss and optimizer, and also a variant with c and d frozen?

  • If I understand correctly, PAGR runs only once at iteration 10k. How many mixtures are reallocated, and how often does the check in Eq. 9 actually fail? Since all amplitudes are nonnegative and the projection is linear, the relative L1 change seems close to the mass fraction (at most zeta = 0.01), which is already below tau_D = 0.05.

  • Which scans pass the stability gate in Sec. 4.4, and what alpha is used for each scan? For the simulated data with exact geometry, I guess alpha is always 0.5.

  • Results with multiple seeds, runtime and memory, a budget sweep under sparse views (25 or 50 views), and comparison with fixed-budget strategies such as 3DGS-MCMC or Taming 3DGS adapted to CT would make the paper much stronger.

1 comment
AuthorsOpenReview ↗

Responses to the questions

Thank you very much for pointing out our oversight. Regarding the method section, as noted in "Overview of Revisions and Additions 1," we have revised the descriptions to eliminate redundancy, re-conducted experiments, and verified the experimental data. Furthermore, as mentioned in "Overview of Revisions and Additions 4," we have expanded the scope of our ablation studies and included additional analysis of the optimization process.

A1: We re-conducted the experiment and verified the experimental data.

A2: We have added ablation experiments on shared/independent amplitude A and shared/independent centers to analyze their effects. (Table 4) We also conducted comparative experiments on R2R^2 without ADC in optimization dynamics. Compared with the lifted kernel under the same initialization conditions, without PAGR, amplitude initialization, and adaptive loss function, the Lifted kernel still outperforms R2R^2 without ADC in convergence. (Figure 6)

A3:Yes, PAGR is performed once per training run, at iteration 10,000. The contribution cap ζ=0.01\zeta=0.01 limits the candidates’ cumulative integrated contribution to 1% of the total, rather than selecting 1% of the mixtures. Consequently, a relatively large number of low-contribution mixtures can be reallocated. Across the 40 runs covering all ten volumes at 50 and 180 views under the default and highest tested budgets, the reallocated mixtures accounted for an average of approximately 13.1% and 8.7% of all mixtures, respectively. All 40 runs passed the check in Eq. 9 on the first attempt with τD=0.05\tau_D=0.05. When we replayed the same checkpoints with the tighter threshold τD=0.01\tau_D=0.01, 18 of the 40 cases failed the initial check, triggering one halving of ζ\zeta.

A4: For the four cases from TCIA, the values ​​are generally around 0.5. In the AAPM dataset, the loss function weights for L192 reach 0.6685 and 0.6179. In the FIPS dataset, the value for "pine" is 0.5/0.5 at the 25 view, while it is higher than this at other angles; at the 75 and 180 views, the values ​​for all three FIPS cases increase to 0.8.

A5: As analyzed in "Overview of Revisions and Additions 2, 4", we have expanded the data analysis to include training duration and peak GPU memory usage, and added comparative curves for different Gaussian kernel budgets under 50 views. Due to time constraints, we were unable to conduct multiple sub-experiments for all trials and adapted experiments with 3DGS-MCMC or Taming 3DGS.

Reviewer_ibTTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes a lifted Gaussians for CT reconstruction. It introduces a extra parameter to the original 3D Gaussians, allowing it to move the Gausians, changing their amplitude. It claims to eliminate the dependency of discrete adaptive control, which most Gaussian splating-based methods use to split, clone and prune the Gaussians with hard thresholds. This work also proposes to set a exact spatial-kernel budget, such that it requires less primitives then the standard ones. Experiments show that it performs better than R2Gaussian on various setups and the performance improvement is larger with more Gaussian primitives.

Critical strengths

The paper is clearly written and easy to follow.

The idea of introducing extra parameter for Gaussians is interesting. The 4D form of Gaussian seems to bring some benefits.

Critical weaknesses

4D form of Gaussian representation alreayd exisit, the techincal contribution is rather limited.

The authors claim that the disadvantages of current density control is its "hard-threshold cloning, splitting, and pruning decisions", and the proposed method "“without reliance on hyperparameter thresholds”. However, the method also relies strongly on thresholding. First of all, the budget B resource constraint can be interpreted as a form of threshold. Second, adding this extra dimension indeed makes the move and amplitudes adjust more flexible as section 4.1 shows. However, the approaches still relies on thresholds like "low-contribution mixtures using a cumulative fraction ζ\zeta", "allowed projection change τ\tau", "ranking accumulated norms", "accepted score" etc. So in the end this method is not relying less on thresholds, and it doesn't support the claim that they have converted the entire density-control process into a threshold-free, continuously differentiable operation.

The main comparison is very limited in Table 1, the authors have only compared with R2Gaussian, despite having more baselines for comparison in Figure 4.

The evaluation is limited, it mainly focuses on the image quality. Yet for such method with maximum budget, computational aspect (for example time, memory) shall be covered as well.

Questions for authors

It is not clear for me, why this method would achieve better reconstruction performance than R2Gaussian in most cases. It doesn't add any extra information, so with the same number of Gaussian primitives, why this method would perform better than R2Gaussian?

At a fixed c, the results remain an ordinary mixture of B spatial Gaussians. Does the performance come from how the representation is optimized?

The lifted kernel give increase in PSNR, but does it drop the SSIM?

One important contribution in this work is the maximum budget B, maybe the authors should set up a ablation experiments on budget vs performance, to have better understanding on this method?

For Figure 3, how is the computation csots when the number of Gaussians align? Maybe a comparison under same computationl budget would also make sense?

As discussed in the weaknesses, this lifted representation introduces extra paramters and steps, my feeling is that is doesn't automatically garanutee faster optimization or lower total memory. Can the authors elaborate on this?

How is the method robust to those newly introduced thresholds, as this is claimed a big strength?

1 comment
AuthorsOpenReview ↗

Responses to the questions

Thanks for your suggestions! 4DGS does indeed exist; however, our approach interprets the "new dimension" from a different perspective—one that is not tied to time or semantics. While PAGR does introduce a new threshold, this module was specifically designed to address challenges inherent in 3DGS-based CT reconstruction; it operates at a different level than the "lifted kernel" mechanism. In fact, when PAGR is disabled in Lifted-GS, the Gaussian kernels—with their positions and amplitudes adjusted via continuous parameter optimization—remain fixed in number, independent of the discrete addition/deletion decisions made by the ADC mechanism. The performance of the model with PAGR disabled can be observed in the ablation studies (Table 3).

We have expanded the content of the table to include not only the number of Gaussian kernels but also comparisons of training time and VRAM usage.(Table 1, 2, 5, C.1, C.2, E.1) As analyzed in "Overview of Revisions and Additions 2" Lifted-GS offers significant advantages in terms of training time and VRAM consumption when operating with a higher Gaussian budget.

A1: Lifted-GS introduces a parameterization of 3DGS that alters its optimization trajectory. Since the reconstruction tasks addressed in our experiments are inherently ill-posed, the specific optimization path taken significantly influences the final converged result. The "lifted kernel" parameterization establishes interdependencies among local Gaussians; because the representation of an object in 3DGS is tied to the joint configuration of its constituent Gaussians, optimizing this joint representation requires coordinating multiple parameters across various Gaussians—parameters that often have conflicting optimization directions. The introduction of "shared planes"—defined by components cc and AA—allows for the grouping of local Gaussians. This mitigates the inconsistency in gradient-based optimization directions, improves the optimization process, and enhances the representation efficiency of the Gaussian clusters. To validate these advantages, as noted in the Overview of Revisions and Additions 4, we have expanded our analysis to include aspects such as optimization dynamics and kernel size. The experimental results demonstrate that this parameterization approach is indeed beneficial to the optimization process. (Figure 6, lines 486-500) (Table 4)

A2: As A1 said, yes.

A3: Overall, Lifted-GS demonstrates advantages in both metrics; however, based on current ablation study results, this phenomenon specifically occurs when only kernel parameterization is employed.

A4: Our performance comparisons for the same test cases under varying budgets should illustrate the relationship between budget and performance. Furthermore, we have included experiments on runtime and computational overhead, allowing for a more comprehensive assessment of the gains and costs associated with different Gaussian budgets. (Figure 20-28, Table 5, E.1)

A5: As mentioned in our response to A6, we have included data regarding training time. When comparing the best results achieved by the two methods under comparable training durations, it is clear that Lifted-GS holds a significant advantage. For example, in terms of the SSIM metric, the best result for L109 outperforms that of R2R^2 (Figure 23); notably, Lifted-GS requires less than 50 minutes of training time, whereas R2R^2 requires 70 minutes (Figure 27). In terms of comparable training time, the R2R^2 with a threshold of 2e-5—while requiring slightly less time than Lifted-GS—exhibits a much larger performance gap. With a higher Gaussian kernel budget, the peak VRAM usage of Lifted-GS remains lower than that of R2R^2.

A6: (Table 1, 2, 5, C.1, C.2, E.1) As analyzed in "Overview of Revisions and Additions 2" Lifted-GS offers significant advantages in terms of training time and VRAM consumption when operating with a higher Gaussian budget.

A7: We have revised the wording in the text to clarify that the "lifted" parameterization and PAGR represent modules at different levels. Due to time constraints, we conducted only a limited number of experiments regarding the threshold robustness of PAGR. We tested the thresholds for fixed candidate contributions and projection variations across ten datasets, each with 50 viewpoints. Tightening the projection variation threshold from 5% to 1% resulted in an average change of −0.017 dB in PSNR and −0.00021 in foreground axial SSIM. Overall, there was no significant impact.

Reviewer_eDgGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper proposes to use a 4D gaussian as opposed to 3D, in order to replace the traditional 3DGS splitting, cloning and pruning behaviour with a continuously differentiable motion trajectory, thus removing the reliance on hyper-parameter tuning for densification optimization. This is appleid in the context of CT reconstruction. They validate this on 7 CT volumes (4 medical, 3 regular objects), and under a constraint of a fixed number of gaussians, are able to achieve slight improvements in PSNR and SSIM (0.16 dB and 0.0011 respectively).

Critical strengths

The motivation is good - removing the need to spend endless time refining the hyper-parameters to prune, clone and split gaussians appropriately. It also ensures that under different gaussian number budgets, or in different scenes/CT volumes, this process does not have to be repeated.

The paper's idea to utilize a dimension-elevated Gaussian is interesting, however it is not well explained.

Critical weaknesses
  1. An explanation of how the paper's proposed method is different from the other Gaussian allocation and budgets (Lines 100-107) methods cited is needed.
  2. Crucially, the method section is extremely difficult to follow, at least for an average ICLR reader. This needs rewriting. It would also help if figures are referenced and used in the text to aid explanation. For example Figure 1 is not cited anywhere in the main body of the paper, yet could be a useful asset to better explain the proposed method. The bottom row of Figure 2 is also unclear to me. If the plane is slicing the 3D gaussian, ie producing a 2D ellipse, then why does the bottom row also contain 3D gaussians?
  3. The quantitative improvement from the proposed method are very marginal - 0.16 dB average PSNR improvement and 0.0011 SSIM improvement in Table 1, the main results table. Presumably utilising 4D rather than 3D gaussians also incurs additional memory and computation cost, yet there is no mention of this. Even if it does take longer, it would be useful to know so that readers can determine whether then accuracy-time trade off is worth it. I could also not find the number of gaussians used in Table 1, which is important given that Table 2 ablates between the number of Gaussians and accuracy. Clearly as more gaussians are used their method performs better, so the results of Table 1 need to be contextualised in this.
  4. As mentioned before, Table 2 and Table 3 are also not cited in the text. The title of Figure 4 "hnsc" is also unclear. There are also a few typos (e.g. L161, L189)
Questions for authors

Please see the points and questions raised in the Weakness section.

1 comment
AuthorsOpenReview ↗

Responses to the questions

Thanks for your suggestion! As stated in the "Overview of Revisions and Additions," we have included additional experimental results and analyses, examining the advantages of our method—and the underlying reasons for these advantages—from the perspectives of optimization dynamics and parameter-sharing ablation.

A1: We have expanded the "Related Work" section, specifically highlighting the differences between the methods in the cited literature and our own approach.(Lines 85)

A2: We sincerely apologize for the lack of clarity in the methodology section. As mentioned in our collective response, we have restructured this section and revised figures to improve their readability; the text has also been updated to provide corresponding with figures.(Lines 74-76, Lines 168-169, Lines 172-176, Lines 196-198) Regarding Figure 2, the z-axis represents the amplitude of the projected 2D Gaussian at specific spatial locations; we have now added the z-axis to the figure and clarified its physical meaning.

A3: We have expanded the content of the table to include not only the number of Gaussian kernels but also comparisons of training time and VRAM usage.(Table 1, 2, C.1, C.2, E.1) As analyzed in "Overview of Revisions and Additions 2" Lifted-GS offers significant advantages in terms of training time and VRAM consumption when operating with a higher Gaussian budget.

A4: We sincerely apologize for this oversight in our writing. We have revised the results section, replaced all instances of "hnsc" with "Head and neck" (consistent with the rest of the manuscript), and thoroughly checked the text for any other potential spelling or similar errors.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overview of Revisions and Additions

We are very grateful to the reviewers for their patience and valuable suggestions. We will provide a response to every point raised by each reviewer. Before addressing these, we would first like to provide an overview of the revisions and additions made to the manuscript:

  1. We have corrected several errors in paper, especially the methods (Section 4): ① Ai1A_{i1} and Ai2A_{i2} in the lifted kernel are shared; we have updated the description in the methodology section accordingly. ② Regarding the redundancy of bikb_{ik}—which we had overlooked during our initial derivation and implementation—we thank the reviewers for pointing this out. We have adjusted the formulation such that the auxiliary mean bikb_{ik} is set to zero and excluded from the optimization process; we have also revised the method section and re-run the experiments to reflect these changes.

  2. Supplement the experimental data and analysis regarding the training time and memory consumption of Lifted-GS. (Table 2) (Figure 24-28) (Table E.1, Lines 1477-1510) Thank you all for your suggestions; they have been invaluable to our work! We have added data on the training time and memory consumption of Lifted-GS. In terms of memory usage, our optimization method introduces additional parameters, which leads to an increase in peak memory consumption during training. However, at higher Gaussian budgets, the peak memory footprint of R^2 is even higher.(Table E.1, Lines 1477-1510) In terms of training time, Lifted-GS is slower than R^2 when the Gaussian budget is low, due to the increased number of parameters; conversely, when the Gaussian budget is higher, Lifted-GS is significantly faster. (Figure 24-28)When compared at the optimal Gaussian budget for each method, we demonstrate a significant advantage in training speed. (Table 2, Lines 330-336) We attribute this to the fact that Lifted-GS tends to produce a denser distribution of smaller-sized Gaussian primitives. (Figure 3) This substantially reduces the computational load for rasterization and voxelization—lowering both the processing overhead and VRAM usage—an advantage that becomes particularly pronounced as the number of Gaussian kernels increases. In short, the optimization path taken by Lifted-GS arrives at a joint representation of Gaussian primitives that differs from that of R^2—a representation that is, from a performance standpoint, better suited to the processes of rasterization and voxelization.

  3. We have added three new, more challenging reconstruction experiments, using data derived from phantoms featured in the AAPM Low-Dose Reconstruction Challenge. These experiments serve to further demonstrate the characteristics and robustness of different reconstruction methods. (Section 5 and Figure 11-13, 17-19)

  4. To further substantiate our findings, we have incorporated additional experimental data and analyses. These additions provide a more comprehensive understanding of the optimization process and the impact of various design choices (Lines 375-481): ① We expanded the scope of the ablation study, explicitly detailing the experimental scenarios employed (Lines 463). ② We supplemented the analysis with a comparative study across varying computational budgets for 50 angles (Figure 20-22). ③ We added an experimental analysis of the optimization trajectory following parameterization of Lifted-GS. A comparative assessment of convergence curves was conducted for four methods – Lifted kernel, Lifted-GS, R2R^2, and R2R^2 without ADC – across two datasets and under varying kernel counts (Figure 6, lines 432-436). The convergence curve shows that R2R^2 exhibits persistent fluctuations during the intermediate stage of training, with the overall curve ceasing to decline; this is attributed to the densification of the ADCs.The remaining three methods demonstrate more rapid convergence. While Lifted-GS exhibits a transient fluctuation at step 10000 due to PAGR reallocation, it ultimately achieves a lower convergence point, followed closely by Lifted kernel and R2R^2 without ADC. ④ Table 4 within the ablation further analyzes the impact of independent versus shared components on performance and kernel shape through the lens of an ablation study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.