- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces FactorSplat, a novel appearance-controllable Gaussian Splatting representation for medical volume rendering. The method aims to leverage pre-trained Gaussian proxies to approximate the computationally expensive Cinematic Volume Rendering, while enabling direct application of unseen Transfer Functions (TFs) defined for different tissue regions during inference—without requiring retraining for each new TF.
Specifically, the authors first initialize Gaussians based on original CT/MR volumetric data and their segmentation labels, extracting material and intensity information from up to 27 local voxels per Gaussian to construct a fixed Material–Intensity Descriptor. The method then employs two complementary branches to model TF variations: a Physical Lookup branch computes RGBA offsets corresponding to TF changes based on local material descriptors; a Functional Encoder branch uses a shared MLP along with low-rank factors for each Gaussian to learn additional appearance responses induced by TF variations.
Experiments were conducted on seven CT/MR datasets, evaluating performance in validation, interpolation, unseen composition, and out-of-distribution (OOD) TF editing. Results show that FactorSplat achieves improvements over Region-aware VEG in terms of average PSNR and local TF editing error, while enabling efficient interactive rendering through caching of TF-conditioned appearances.
Critical strengths
-
The research question is well-defined, and a reasonable method design is proposed to address the limitations of existing Gaussian Proxies in terms of TF editability. Compared to existing editable Gaussian volume visualization methods such as VEG and iVR-GS, this paper offers certain methodological value through the integration of region-specific functional TF conditioning and cinematic rendering supervision.
-
The experiment covered various unseen TF editing scenarios and verified some key designs through ablation experiments. This paper not only evaluates TF interpolation within the training range, but also considers the unseen combinations of observed editing operations, as well as the more challenging OOD TF editing. This enables a certain degree of reflection of the method's generalization ability to new TFs.
Critical weaknesses
- The consistency between the Local Material-Intensity Descriptor and the optimized Gaussian Geometry has not been adequately verified, which may affect the accuracy of tissue-level TF editing.
In Section 3.2 (Lines 181–197), the author extracts up to 27 local voxels for each Gaussian, records its Region ID, Intensity Bin and normalized weight, and uses this information for subsequent TF conditioning. However, according to Section 3.4 (Lines 274–284, Eq. 12), these sampling weights are calculated based on the center and covariance of the Gaussian in the initialization stage. Although the position, scale, and covariance of the Gaussian will be optimized during subsequent training, the corresponding Region–Intensity Identifiers and Weights remain constant. For Clone and Split operations, the new Gaussian directly inherits the material descriptors of the existing Gaussian instead of recalculating based on the new geometry. This design has not been adequately discussed regarding how it maintains the stability of material associations during the training process and how the optimized Gaussian spatial coverage range is consistent with the fixed material descriptors. For instance, a Gaussian initially mainly covers the vascular area during initialization. However, after position adjustment or scale change, its final coverage range may include more soft tissues. At this point, when the user modifies the vascular TF, this Gaussian still responds according to the vascular material weights set during initialization, rather than based on the actual tissue distribution covered by the final Gaussian.
- The experiment has not sufficiently distinguished the independent contributions of each method component, nor has it fully demonstrated the trade-off in terms of quality and efficiency compared to the standard volume rendering.
Although Table 2 has provided ablation experiments for Physical Lookup and Learned Residual, the Coverage-aware Pruning proposed in Section 3.4 lacks independent ablation verification. Moreover, the impact of local voxel sampling window size, Gaussian Descriptor support range, and the number of GS(looks like FactorSplat uses more GS than VEG in Table 5 in Appendix) on generalization performance is still unclear.
- Another concern is that one of the main motivations of this paper is to replace the computationally expensive Cinematic Volume Rendering with Gaussian Proxy. However, the experiments mainly compared with the existing Gaussian-based VEG methods, and did not report the rendering performance, memory consuming of the original Cinematic Volume Path Tracer used to generate the Ground Truth.
Especially, based on my understanding, volume rendering developped for decades, there are a lot mature implemnetation that can perform efficiently, I think the author should compare the mature volume rendering with their two rendering protocol(Fast render and quality render) .
Questions for authors
-
Could the author provide a more detailed explanation of the specific implementation of Gaussian Initialization, including the distribution of the initial Gaussian's position, the Grid Spacing, and how the Covariance is determined? Since the Region-Intensity Descriptor and its weights remain constant after initialization, will these geometric changes lead to inconsistencies between the final Gaussian Support and the local material descriptor? If the Descriptor is recalculated using the optimized Gaussian Geometry, or if the Material–Intensity Weights are updated periodically during the training process, will it be possible to enhance the TF Editing Fidelity near the tissue boundaries?
-
Could the author provide the PSNR corresponding to the Cached Fast Renderer, as well as the FPS corresponding to the Quality Renderer, so that the rendering quality and rendering speed can be compared under the same operating configuration?
-
In the introduction, the authors introduced Gauss because this representation is efficient. The author merely believes that using rasterization for rendering is more efficient? Do you still think that this efficiency can be expressed as GS can be more compact than volume voxel (which represents an efficient expression)? But in this paper, it looks like for each GS primitive, we need to attatch 27 samples for it? Could author provide the memory consuming comparison for different methods?
-
I am curious about the comparison with the standard volume rendering implemention with GPU acceleration. I feel like this paper use the GS to get efficient rendering. But could the author tell me how to claim that FactorSplat is a good way to represent the scene by replacing the volume representation? Could you provide the memory consuming by different voxel resolution for a scene compared with FactorSplat? At the same time, FactorSplat lose some rendering fidelity to increase the rendering speed. Could you provide the comparison that if we modify some parameters for volume rendering(like sampleing steps in ray marching) to increase the rendering speed but degrade the rendering quality? I hope the authors can convince me that FactorSplat is efficent representation in memory consuming and rendering speed than prevous representation(NeRF, Voxel, etc.)
3 comments
Response to Reviewer cQ2G
Q2. PSNR of the cached fast renderer and FPS of the quality renderer
The fast renderer is 2.8× faster than the quality renderer and loses 0.004 dB. Both are measured on the same 56 views (8 validation views for each of the seven scans, 1600 × 1600; Table 3). The names say how a path renders, not how good its image is: "quality" recomputes the appearance every frame with the rasterizer used in training, "cached" computes it once per TF, and "fast" adds the speed-optimized rasterizer.
| Renderer | PSNR (dB) | FPS, B200 | FPS, A100 |
|---|---|---|---|
| Training rasterizer ("quality") | 35.20 | 201.6 | 134.9 |
| + cached appearance ("cached quality") | 35.20 | 229.5 | 150.1 |
| + fast rasterizer ("cached fast") | 35.20 | 557.6 | 522.1 |
Over all 4,760 held-out images the mean loss is 0.0008 dB. The 524 FPS and 1.17 ms of the submission are unchanged (Table 1). Table 3 is a separate run that pairs quality and speed, and it measures 557.6 FPS and 1.10 ms on B200.
Q3. Is the Gaussian representation more compact than voxels, given 27 samples per primitive?
FactorSplat is 2 to 19 times smaller than the scan and label volumes it is built from. The efficiency we claim is rendering speed with an editable TF, not storage. Storage per scan (Table 9):
| Representation | Size (MiB) |
|---|---|
| FactorSplat | 32.4–113.7 (mean 81.4) |
| FactorSplat with cached appearance for all 41 TFs | mean 107.1 |
| Scan and label volume, native resolution | 237.8–1,062.0 |
| Scan and label volume, half resolution per axis | 29.8–133.2 |
A Gaussian takes 423 bytes: 132 for geometry and base appearance, 128 for the low-rank factors, and 163 for the 27 samples (IDs, weights and a count). The descriptor is 39% of the model, and it is what lets a TF edit update the appearance without touching the geometry. A voxel volume reaches a similar size at half resolution per axis, which is one eighth of the voxels. The FactorSplat sizes are uncompressed 32-bit tensors; the quantization and compression methods used for Gaussian splats apply to them and were not used here.
W3 and Q4. Performance and memory of the cinematic path tracer; comparison with standard GPU volume rendering
At its fastest setting the reference path tracer is 25 times slower than FactorSplat and 1.71 dB less accurate. At the tested budget of 16 it is more accurate, at about 140 times FactorSplat's frame time (Table 10). This is the experiment you describe. The path tracer's quality parameter is the number of accumulated frames, which is 1000 for our reference images. We lowered it, kept the denoiser on, and rendered the same 56 views as in Q2 on the same A100:
| Renderer | PSNR (dB) | Time per frame |
|---|---|---|
| Path tracer, budget 64 | 39.52 | 921 ms |
| Path tracer, budget 16 | 36.52 | 268 ms |
| Path tracer, one render call | 33.48 | 49.5 ms |
| FactorSplat, quality | 35.20 | 8.03 ms |
| FactorSplat, cached fast | 35.20 | 1.95 ms |
The comparison favours the path tracer in two ways: PSNR is measured against its own converged output, and its times exclude readback and display.
Memory. FactorSplat's peak allocation is 244–452 MiB with the cached fast renderer and 732–2,718 MiB with the quality renderer. The path tracer used up to 2,170 MiB of GPU memory in these runs. Storage against the voxel volume at two resolutions is in the Q3 table.
Standard volume rendering and other representations. The path tracer is the standard renderer for these images. A classical emission–absorption ray caster does not compute their global illumination, so it renders a different image and is not a baseline for this target. The voxel representation is covered by Table 10 (speed and quality) and by the Q3 table (storage). We did not add a NeRF baseline; the Gaussian methods of Table 1 are the closest prior work on editable proxies. In short, FactorSplat gives the path-traced look at about 2 ms per frame with an editable TF, where the path tracer needs 49.5 ms for a less accurate image and 268 ms for a more accurate one. The discussion section states this.
Weak acceptance
Thanks for the experiments.
Response to Reviewer cQ2G
Thank you for the detailed questions. Each one is answered with a new measurement in the revised PDF. In short: re-computing the descriptors during training, as you suggest, improves heart by up to 3.12 dB and puts it ahead of VEG in every split; coverage-aware pruning now has its own ablation; the fast renderer loses 0.004 dB on the same views; and the reference path tracer at its fastest setting is 25 times slower than FactorSplat and less accurate.
All numbers refer to the revised PDF. Tables 3–5 of the submission are now Tables 4–6, and the new Table 3 is the paired renderer comparison.
W1 and Q1. Initialization, and consistency between the fixed descriptor and the optimized geometry
Initialization (Sec. 3.4, Table 7). Six scans start from Gaussians predicted from the volume, and keep the predicted centres, anisotropic scales and rotations, so they have no common grid spacing. Hand starts from a regular grid of 120,000 isotropic Gaussians with identity rotation. Sec. 3.4 now describes both initializations and Table 7 lists every scan; the submission described only the grid. The covariance is always built from the stored scale and rotation. The initial counts range from 5,399 (knee) to 1,794,612 (intestine). The descriptor is attached afterwards: the 3 × 3 × 3 voxels around each centre give up to 27 (region, intensity bin) samples, weighted by the initial Gaussian (Eq. 11).
Does re-computing the descriptor from the optimized geometry help? Clearly on heart, moderately on knee, and not on hand. We ran the experiment you propose: after each densification step, sample IDs and weights are re-computed from the current centre and covariance (Table 8). Change from fixed to re-computed descriptors:
With re-computed descriptors, heart is ahead of VEG in all four splits, in both PSNR and edit error (VEG: 28.79 / 28.21 / 27.80 / 25.70 dB and 1.22 / 2.26 / 3.03 / 5.93). Knee improves in all four splits and both metrics. Hand is neutral on three splits and loses 0.58 dB on OOD. The fixed descriptor therefore limits editing on some scans and not on others, so we report re-computation as a control (one seed per run) and keep the submitted method and its results unchanged. The two knee runs were trained in a different software environment from the submission and are compared with each other.
W2. Pruning ablation, sampling window, descriptor support and number of Gaussians
Coverage-aware pruning now has its own ablation (Table 8). Only the pruning rule changes: opacity under the base TF instead of the maximum opacity over the training TFs.
The rule keeps Gaussians that are transparent under the base TF but visible under another training TF. Without it the final model has 27% and 15% fewer Gaussians and a lower PSNR on every split.
Number of Gaussians. FactorSplat's lead over VEG is much larger than the effect of the count in our ablation. The ablation above removes 27% of the heart Gaussians and costs 0.15–0.25 dB, a fraction of FactorSplat's 0.9–2.0 dB lead over VEG on heart validation, interpolation and composition. On knee FactorSplat has the fewer Gaussians (80.4k against 82.7k); on the other six scans it has more (for example 278.6k against 146.3k on heart). Both methods use their own density control, and Table 6 reports the counts.
Window and support. The window is 3 × 3 × 3 voxels, one voxel on each side of the centre, in all experiments. The re-computation control above tests the related question of whether the support must follow the geometry.