acceptodds
Under review as a conference paper at ICLR 2027

FactorSplat: Appearance-Controllable Gaussian Proxies for Medical Volume Rendering

Abstract

Transfer functions (TFs) control color and visibility in medical volume rendering, but image-trained Gaussian proxies typically bake one transfer function into their appearance. We present FactorSplat, a per-scene N-dimensional Gaussian splatting (N-DGS) proxy that accepts region-specific intensity-to-RGBA curves at inference. A local lookup applies the authored color and opacity change, while a shared functional encoder and low-rank per-Gaussian factors learn the residual appearance response. Geometry and directional appearance remain shared across presets, with visibility control and TF-aware pruning preserving the ability to hide and reveal structures. On seven CT and MR scans, FactorSplat improves mean PSNR and changed-region error over region-aware VEG across validation, interpolation, unseen composition, and out-of-distribution (OOD) edits. Across these four splits, seven-scan mean PSNR gains over VEG range from 1.10 to 1.52 dB. One checkpoint per scan supports unseen edits without retraining. At 1600², the cached fast renderer averages 524 FPS with 1.17 ms TF switches.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Gaussian splatting, medical volume rendering, transfer functions, conditional representations, neural rendering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cQ2GOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces FactorSplat, a novel appearance-controllable Gaussian Splatting representation for medical volume rendering. The method aims to leverage pre-trained Gaussian proxies to approximate the computationally expensive Cinematic Volume Rendering, while enabling direct application of unseen Transfer Functions (TFs) defined for different tissue regions during inference—without requiring retraining for each new TF.

Specifically, the authors first initialize Gaussians based on original CT/MR volumetric data and their segmentation labels, extracting material and intensity information from up to 27 local voxels per Gaussian to construct a fixed Material–Intensity Descriptor. The method then employs two complementary branches to model TF variations: a Physical Lookup branch computes RGBA offsets corresponding to TF changes based on local material descriptors; a Functional Encoder branch uses a shared MLP along with low-rank factors for each Gaussian to learn additional appearance responses induced by TF variations.

Experiments were conducted on seven CT/MR datasets, evaluating performance in validation, interpolation, unseen composition, and out-of-distribution (OOD) TF editing. Results show that FactorSplat achieves improvements over Region-aware VEG in terms of average PSNR and local TF editing error, while enabling efficient interactive rendering through caching of TF-conditioned appearances.

Critical strengths
  1. The research question is well-defined, and a reasonable method design is proposed to address the limitations of existing Gaussian Proxies in terms of TF editability. Compared to existing editable Gaussian volume visualization methods such as VEG and iVR-GS, this paper offers certain methodological value through the integration of region-specific functional TF conditioning and cinematic rendering supervision.

  2. The experiment covered various unseen TF editing scenarios and verified some key designs through ablation experiments. This paper not only evaluates TF interpolation within the training range, but also considers the unseen combinations of observed editing operations, as well as the more challenging OOD TF editing. This enables a certain degree of reflection of the method's generalization ability to new TFs.

Critical weaknesses
  1. The consistency between the Local Material-Intensity Descriptor and the optimized Gaussian Geometry has not been adequately verified, which may affect the accuracy of tissue-level TF editing.

In Section 3.2 (Lines 181–197), the author extracts up to 27 local voxels for each Gaussian, records its Region ID, Intensity Bin and normalized weight, and uses this information for subsequent TF conditioning. However, according to Section 3.4 (Lines 274–284, Eq. 12), these sampling weights are calculated based on the center and covariance of the Gaussian in the initialization stage. Although the position, scale, and covariance of the Gaussian will be optimized during subsequent training, the corresponding Region–Intensity Identifiers and Weights remain constant. For Clone and Split operations, the new Gaussian directly inherits the material descriptors of the existing Gaussian instead of recalculating based on the new geometry. This design has not been adequately discussed regarding how it maintains the stability of material associations during the training process and how the optimized Gaussian spatial coverage range is consistent with the fixed material descriptors. For instance, a Gaussian initially mainly covers the vascular area during initialization. However, after position adjustment or scale change, its final coverage range may include more soft tissues. At this point, when the user modifies the vascular TF, this Gaussian still responds according to the vascular material weights set during initialization, rather than based on the actual tissue distribution covered by the final Gaussian.

  1. The experiment has not sufficiently distinguished the independent contributions of each method component, nor has it fully demonstrated the trade-off in terms of quality and efficiency compared to the standard volume rendering.

Although Table 2 has provided ablation experiments for Physical Lookup and Learned Residual, the Coverage-aware Pruning proposed in Section 3.4 lacks independent ablation verification. Moreover, the impact of local voxel sampling window size, Gaussian Descriptor support range, and the number of GS(looks like FactorSplat uses more GS than VEG in Table 5 in Appendix) on generalization performance is still unclear.

  1. Another concern is that one of the main motivations of this paper is to replace the computationally expensive Cinematic Volume Rendering with Gaussian Proxy. However, the experiments mainly compared with the existing Gaussian-based VEG methods, and did not report the rendering performance, memory consuming of the original Cinematic Volume Path Tracer used to generate the Ground Truth.

Especially, based on my understanding, volume rendering developped for decades, there are a lot mature implemnetation that can perform efficiently, I think the author should compare the mature volume rendering with their two rendering protocol(Fast render and quality render) .

Questions for authors
  1. Could the author provide a more detailed explanation of the specific implementation of Gaussian Initialization, including the distribution of the initial Gaussian's position, the Grid Spacing, and how the Covariance is determined? Since the Region-Intensity Descriptor and its weights remain constant after initialization, will these geometric changes lead to inconsistencies between the final Gaussian Support and the local material descriptor? If the Descriptor is recalculated using the optimized Gaussian Geometry, or if the Material–Intensity Weights are updated periodically during the training process, will it be possible to enhance the TF Editing Fidelity near the tissue boundaries?

  2. Could the author provide the PSNR corresponding to the Cached Fast Renderer, as well as the FPS corresponding to the Quality Renderer, so that the rendering quality and rendering speed can be compared under the same operating configuration?

  3. In the introduction, the authors introduced Gauss because this representation is efficient. The author merely believes that using rasterization for rendering is more efficient? Do you still think that this efficiency can be expressed as GS can be more compact than volume voxel (which represents an efficient expression)? But in this paper, it looks like for each GS primitive, we need to attatch 27 samples for it? Could author provide the memory consuming comparison for different methods?

  4. I am curious about the comparison with the standard volume rendering implemention with GPU acceleration. I feel like this paper use the GS to get efficient rendering. But could the author tell me how to claim that FactorSplat is a good way to represent the scene by replacing the volume representation? Could you provide the memory consuming by different voxel resolution for a scene compared with FactorSplat? At the same time, FactorSplat lose some rendering fidelity to increase the rendering speed. Could you provide the comparison that if we modify some parameters for volume rendering(like sampleing steps in ray marching) to increase the rendering speed but degrade the rendering quality? I hope the authors can convince me that FactorSplat is efficent representation in memory consuming and rendering speed than prevous representation(NeRF, Voxel, etc.)

3 comments
AuthorsOpenReview ↗

Response to Reviewer cQ2G

Thank you for the detailed questions. Each one is answered with a new measurement in the revised PDF. In short: re-computing the descriptors during training, as you suggest, improves heart by up to 3.12 dB and puts it ahead of VEG in every split; coverage-aware pruning now has its own ablation; the fast renderer loses 0.004 dB on the same views; and the reference path tracer at its fastest setting is 25 times slower than FactorSplat and less accurate.

All numbers refer to the revised PDF. Tables 3–5 of the submission are now Tables 4–6, and the new Table 3 is the paired renderer comparison.

W1 and Q1. Initialization, and consistency between the fixed descriptor and the optimized geometry

Initialization (Sec. 3.4, Table 7). Six scans start from Gaussians predicted from the volume, and keep the predicted centres, anisotropic scales and rotations, so they have no common grid spacing. Hand starts from a regular grid of 120,000 isotropic Gaussians with identity rotation. Sec. 3.4 now describes both initializations and Table 7 lists every scan; the submission described only the grid. The covariance is always built from the stored scale and rotation. The initial counts range from 5,399 (knee) to 1,794,612 (intestine). The descriptor is attached afterwards: the 3 × 3 × 3 voxels around each centre give up to 27 (region, intensity bin) samples, weighted by the initial Gaussian (Eq. 11).

Does re-computing the descriptor from the optimized geometry help? Clearly on heart, moderately on knee, and not on hand. We ran the experiment you propose: after each densification step, sample IDs and weights are re-computed from the current centre and covariance (Table 8). Change from fixed to re-computed descriptors:

ScanValidationInterpolationCompositionOOD
Heart, PSNR (dB)+0.36+1.01+1.52+3.12
Heart, edit error ×1001.32 → 1.092.62 → 2.033.17 → 2.369.78 → 5.87
Hand, PSNR (dB)+0.04−0.10+0.10−0.58
Hand, edit error ×1000.86 → 0.841.52 → 1.540.97 → 0.943.81 → 4.18
Knee, PSNR (dB)+0.15+0.15+0.56+0.64
Knee, edit error ×1001.31 → 1.221.43 → 1.361.79 → 1.5411.66 → 10.73

With re-computed descriptors, heart is ahead of VEG in all four splits, in both PSNR and edit error (VEG: 28.79 / 28.21 / 27.80 / 25.70 dB and 1.22 / 2.26 / 3.03 / 5.93). Knee improves in all four splits and both metrics. Hand is neutral on three splits and loses 0.58 dB on OOD. The fixed descriptor therefore limits editing on some scans and not on others, so we report re-computation as a control (one seed per run) and keep the submitted method and its results unchanged. The two knee runs were trained in a different software environment from the submission and are compared with each other.

W2. Pruning ablation, sampling window, descriptor support and number of Gaussians

Coverage-aware pruning now has its own ablation (Table 8). Only the pruning rule changes: opacity under the base TF instead of the maximum opacity over the training TFs.

Without coverage-aware pruningValidationInterpolationCompositionOODGaussians
Heart, PSNR change (dB)−0.16−0.25−0.15−0.19278,148 → 201,943
Hand, PSNR change (dB)−0.04−0.05−0.05−0.07135,598 → 115,541

The rule keeps Gaussians that are transparent under the base TF but visible under another training TF. Without it the final model has 27% and 15% fewer Gaussians and a lower PSNR on every split.

Number of Gaussians. FactorSplat's lead over VEG is much larger than the effect of the count in our ablation. The ablation above removes 27% of the heart Gaussians and costs 0.15–0.25 dB, a fraction of FactorSplat's 0.9–2.0 dB lead over VEG on heart validation, interpolation and composition. On knee FactorSplat has the fewer Gaussians (80.4k against 82.7k); on the other six scans it has more (for example 278.6k against 146.3k on heart). Both methods use their own density control, and Table 6 reports the counts.

Window and support. The window is 3 × 3 × 3 voxels, one voxel on each side of the centre, in all experiments. The re-computation control above tests the related question of whether the support must follow the geometry.

AuthorsOpenReview ↗

Response to Reviewer cQ2G

Q2. PSNR of the cached fast renderer and FPS of the quality renderer

The fast renderer is 2.8× faster than the quality renderer and loses 0.004 dB. Both are measured on the same 56 views (8 validation views for each of the seven scans, 1600 × 1600; Table 3). The names say how a path renders, not how good its image is: "quality" recomputes the appearance every frame with the rasterizer used in training, "cached" computes it once per TF, and "fast" adds the speed-optimized rasterizer.

RendererPSNR (dB)FPS, B200FPS, A100
Training rasterizer ("quality")35.20201.6134.9
+ cached appearance ("cached quality")35.20229.5150.1
+ fast rasterizer ("cached fast")35.20557.6522.1

Over all 4,760 held-out images the mean loss is 0.0008 dB. The 524 FPS and 1.17 ms of the submission are unchanged (Table 1). Table 3 is a separate run that pairs quality and speed, and it measures 557.6 FPS and 1.10 ms on B200.

Q3. Is the Gaussian representation more compact than voxels, given 27 samples per primitive?

FactorSplat is 2 to 19 times smaller than the scan and label volumes it is built from. The efficiency we claim is rendering speed with an editable TF, not storage. Storage per scan (Table 9):

RepresentationSize (MiB)
FactorSplat32.4–113.7 (mean 81.4)
FactorSplat with cached appearance for all 41 TFsmean 107.1
Scan and label volume, native resolution237.8–1,062.0
Scan and label volume, half resolution per axis29.8–133.2

A Gaussian takes 423 bytes: 132 for geometry and base appearance, 128 for the low-rank factors, and 163 for the 27 samples (IDs, weights and a count). The descriptor is 39% of the model, and it is what lets a TF edit update the appearance without touching the geometry. A voxel volume reaches a similar size at half resolution per axis, which is one eighth of the voxels. The FactorSplat sizes are uncompressed 32-bit tensors; the quantization and compression methods used for Gaussian splats apply to them and were not used here.

W3 and Q4. Performance and memory of the cinematic path tracer; comparison with standard GPU volume rendering

At its fastest setting the reference path tracer is 25 times slower than FactorSplat and 1.71 dB less accurate. At the tested budget of 16 it is more accurate, at about 140 times FactorSplat's frame time (Table 10). This is the experiment you describe. The path tracer's quality parameter is the number of accumulated frames, which is 1000 for our reference images. We lowered it, kept the denoiser on, and rendered the same 56 views as in Q2 on the same A100:

RendererPSNR (dB)Time per frame
Path tracer, budget 6439.52921 ms
Path tracer, budget 1636.52268 ms
Path tracer, one render call33.4849.5 ms
FactorSplat, quality35.208.03 ms
FactorSplat, cached fast35.201.95 ms

The comparison favours the path tracer in two ways: PSNR is measured against its own converged output, and its times exclude readback and display.

Memory. FactorSplat's peak allocation is 244–452 MiB with the cached fast renderer and 732–2,718 MiB with the quality renderer. The path tracer used up to 2,170 MiB of GPU memory in these runs. Storage against the voxel volume at two resolutions is in the Q3 table.

Standard volume rendering and other representations. The path tracer is the standard renderer for these images. A classical emission–absorption ray caster does not compute their global illumination, so it renders a different image and is not a baseline for this target. The voxel representation is covered by Table 10 (speed and quality) and by the Q3 table (storage). We did not add a NeRF baseline; the Gaussian methods of Table 1 are the closest prior work on editable proxies. In short, FactorSplat gives the path-traced look at about 2 ms per frame with an editable TF, where the path tracer needs 49.5 ms for a less accurate image and 268 ms for a more accurate one. The discussion section states this.

Reviewer_cQ2GOpenReview ↗

Weak acceptance

Thanks for the experiments.

Reviewer_2vdjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a method for fast volume rendering for medical data by reconstructing a 3D Gaussian Splatting model. Similar to previous methods, it uses volumetric path tracing to generate images of the volume and 3DGS to reconstruct a model that closely matches the path-traced images. The paper addresses an issue in previous work, in which the transfer function is chosen during the baking step and cannot be changed without a time-consuming re-render and reconstruction.

The authors achieve this by decomposing the appearance model into a transfer-function-dependent part and an independent part, and by generalizing to novel transfer functions by varying them during training, thereby forcing the model to learn the decomposition.

They show that this allows them to change the transfer function at test time with minimal loss in visual quality on multiple datasets.

The main contribution of the paper is the appearance model that allows for the decomposition and therefore a modification of the transfer function during test time

Critical strengths

The general idea of parameterizing the appearance model such that it learns the decomposition during reconstruction is the most critical strength of the paper. This is a promising idea that seems very helpful for the visualization community.

Critical weaknesses
  1. The paper mischaracterizes and does not compare to Render-FM
  • Related Work says Render-FM targets "a fixed rendering appearance". Render-FM actually takes the TF as input and claims dynamic TF modification and generalization to unseen TFs.
  • There is no comparison, and the paper doesn't discuss the differences. Those include re-running inference per TF (a few seconds) against a 1.17 ms switch.
  • This makes the novelty claim ("TF changes without retraining") look wrong. Render-FM takes a learned approach to the problem while the paper presents a hand-crafted solution. They should be compared in detail since they are to orthogonal solutions for the same problem.
  1. The empirical evidence is lacking
  • The qualitative support is weak: Fig. 3 has no zoom-ins or crops, so the differences are hard to see (p8). The ablation claims have no images (p9).
  • Several ablation effects are small or mixed (p7, Table 2). Dropping the gate or the functional code barely moves PSNR. I checked the table, and removing the functional code even improves two vascular columns.
  • The headline speed claim (524 vs 144 FPS) is measured only on B200 GPUs (p9). That doesn't show real-time performance on consumer hardware, which is where 3DGS models are meant to be used. (I could imagine direct volume rendering on a b200 to reach similar performance)
Questions for authors

I like the idea of the paper, and the results presented look good, but I have serious concerns about the paper's soundness (see weaknesses), the writing, and the figures. The language of the paper was hard to follow, and I had to reread many paragraphs to make sense of it. In general I think the paper overuses (often sloppy) math notation for things that could be explained better with text.

Here are my comments on minor issues in the paper:

1. Page 1, line 53

Marked text: "so the TF is no longer an input."

Comment: it is an input. just one that is fixed in the baking step. I find the wording misleading.

2. Page 2, line 59

Marked text: "allocating new parameters."

Comment: allocating what parameters? I dont get this sentence

3. Page 2, line 69-71

Marked text: "Centering the branches on an authored base TF gives exact identity at that preset, and pruning by maximum opacity over the training bank protects primitives needed by different training TFs."

Comment: this is to compressed and hard to understand

4. Page 4, line 169

Marked text: "grid separately {h_s}, s=1..S, by" (S)

Comment: What is S? I assume its the transfer function resolution?

5. Page 5, line 240

Marked text: "FactorSplat uses (b L , b R ) = (1, 1)."

Comment: why is this defined as variables in the first place? I see that you ablate on this but there are so many variables at that point that its hard to keep track.

6. Page 5, line 264

Marked text: "…the unmodified tile rasterizer…"

Comment: the tile rasterizer from 3dgs (kerbl et al)?

7. Page 6, line 304

Marked text: "r = 8, 64 hidden units,"

Comment: here it should be mentioned that this is for the local function encoder (who can remember what r was at that point)

8. Page 6, line 308

Comment: why 0.04? seems arbitrary

9. Page 7, line 342

Marked text: "Table 2: Two-scene ablations (r=8). PSNR↑, mean rank, and worst gap over eight comparisons."

Comment: quality improvements seems to be really minor in a lot of cases. a qualitative comparison (example images) would help a lot here.

10. Page 8, line 429-430

Marked text: "Figure 3: Held-out CT and MR examples. Heart (rows 1–3), vascular (4–5), nose (6), and hand (7). Specialists are trained separately on each target TF. Full-frame PSNR; bold marks each row’s best."

Comment: zoom ins an cout-outs would help a lot since its hard to make out details / differences in the images

11. Page 9, line 435-441

Comment: i think images to support that claim would help a lot here.

12. Page 13, line 650-651

Marked text: "Appearance scales. All scans use lookup scales λ cL = 1 and λ α L = 4, and residual scales λ c R = 0.25 α and λ α R = 4."

Comment: how where theses values chosen?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2vdj

Thank you for the detailed review and the line-by-line comments. The revision describes Render-FM as a TF-conditioned model and compares with it directly: FactorSplat updates a TF in 1.17 ms where Render-FM needs a 2.6–2.8 s forward pass, and has less than half its edit error. The revision also adds ablation images, quality and speed measured on the same views on two GPUs, and a comparison with the reference path tracer at reduced sample budgets.

All numbers refer to the revised PDF. Tables 3–5 of the submission are now Tables 4–6, and the new Table 3 is the paired renderer comparison.

W1. The paper mischaracterizes Render-FM and does not compare to it

The two methods act at different points, and the contribution of FactorSplat stands: a TF edit updates only the colour and opacity of an existing proxy, in 1.17 ms. Render-FM does take the TF as input and handles unseen TFs, and the introduction and related work now say so. As you note, it re-runs inference for every TF. The two methods compare as follows (Render-FM numbers from its paper):

Render-FMFactorSplat
What is trainedone network, on 991 CT scansone proxy per scan (CT or MR), about 10 min
New scanone forward passa new fit from reference renders
How the TF entersper-voxel RGBA input channelsregion curves, read by a lookup and a small shared encoder
New TF on a loaded scananother forward pass that predicts all Gaussians again, 2.6–2.8 s1.17 ms update of colour and opacity, geometry unchanged
Working resolution1.5 mm voxelsnative scan

Render-FM needs no per-scan fit. FactorSplat makes the TF edit itself interactive, three orders of magnitude faster per TF, at the native resolution of the scan and on MR as well as CT. We agree that the two are orthogonal solutions, and the paper now presents them this way.

Learned against hand-crafted. FactorSplat is not a purely hand-crafted solution. It combines both, and both are needed. The lookup applies the authored colour and opacity change exactly. The learned part corrects for shading and occlusion. Removing the learned part costs 0.50–1.60 dB and removing the lookup costs 0.89–3.47 dB, in all 8 scan–split cells of Table 2.

Direct comparison (Table 11, Fig. 6). We ran the pretrained Render-FM model on heart and vascular, with 4 validation TFs × 40 cameras:

ScanEdit error ×100, Render-FM / FactorSplatPSNR (dB), Render-FM / FactorSplat
Heart2.84 / 1.3115.37 / 30.79
Vascular3.76 / 1.6020.89 / 34.20

FactorSplat has less than half the edit error of Render-FM on both scans. Edit error subtracts each model's own base image, so the static differences in lighting and post-processing of our reference renderer, to which the released Render-FM model is not adapted, largely cancel; in full-frame PSNR they do not. The table compares the released general model with a per-scan fit, which is how each method is meant to be used.

AuthorsOpenReview ↗

Response to Reviewer 2vdj

W2. The empirical evidence is lacking

Ablations (Tables 2 and 5). Each component is needed for the case it is built for. The learned residual and the TF lookup carry the appearance change on every TF: removing either lowers PSNR and raises the edit error in all 8 scan–split cells, by 0.50–1.60 dB and 0.89–3.47 dB. The functional code and the visibility gate are built for TFs unlike the training ones, so the OOD columns are where they should be read, and there they are not small:

  • Functional code. Without it, a new TF receives the embedding of its nearest training preset. That is enough when the new TF is close to a training one, which is why the embedding is 0.10 and 0.48 dB ahead on two vascular columns. On the OOD TFs it is 1.02 dB (heart) and 2.84 dB (vascular) behind.
  • Visibility gate (Sec. 3.3). It sets a Gaussian's opacity to exactly zero when the edit hides the tissue that makes up most of it. Only the OOD split contains such an edit, so the other columns are unchanged by construction; on OOD the gate is worth 0.32 and 0.52 dB.

Images. Figure 5 shows zoomed crops and edit-error maps for every ablation row on heart and vascular. On vascular the full model removes the hidden tissue as in the reference, and the same checkpoint with the gate turned off leaves a visible haze around the vessels (30.56 against 35.15 dB in that view).

Speed. Quality and speed are now measured on the same 56 views on two GPUs (Table 3):

RendererPSNR (dB)FPS, B200FPS, A100
Training rasterizer ("quality")35.20201.6134.9
+ fast rasterizer ("cached fast")35.20557.6522.1

The 524 FPS and 1.17 ms of the submission are unchanged (Table 1); the paired run measures 557.6 FPS and a 1.10 ms TF update on the B200, and a 1.25 ms update on the A100. The speed is not a property of the B200: the A100, released in 2020, reaches 522.1 FPS, more than 8 times a 60 FPS budget, and the cached fast renderer allocates at most 452 MiB of GPU memory. We have not timed a consumer GPU.

Direct volume rendering does not reach similar performance (Table 10). We measured the renderer that produces the reference images, on the same A100 and the same 56 views, with its sample budget lowered and its denoiser on:

RendererPSNR (dB)Time per frame
Path tracer, budget 6439.52921 ms
Path tracer, budget 1636.52268 ms
Path tracer, one render call33.4849.5 ms
FactorSplat, cached fast35.201.95 ms

At its fastest setting the path tracer is 25 times slower than FactorSplat and 1.71 dB less accurate. At the tested budget of 16 it is more accurate than FactorSplat, at 268 ms per frame, about 140 times FactorSplat's frame time. The comparison favours the path tracer in two ways: PSNR is measured against its own converged output, and its times exclude readback and display. A classical emission–absorption ray caster does not compute the global illumination of the reference images, so it renders a different image and is not a baseline for this target.

AuthorsOpenReview ↗

Response to Reviewer 2vdj

All locations refer to the revised PDF. Sec. 3 now explains each step in words before its equation, and symbols are defined where they are used.

  1. p1 l53, "so the TF is no longer an input". Reworded: in a proxy fitted to one appearance, the TF is fixed during construction and cannot be edited afterwards. Render-FM and VEG are described separately as methods that do accept a TF.
  2. p2 l59, "allocating new parameters". It meant that a new TF needs neither a new learned preset embedding nor another fitted model. The sentence now says this.
  3. p2 l69–71. Now two separate statements. Both appearance offsets are zero at the base TF, so the base reconstruction is unchanged. Pruning looks at a Gaussian's opacity under all training TFs, so a Gaussian that is transparent under one TF but visible under another is kept.
  4. p4 l169, S. Yes, S is the resolution of the exported TF curve. It is now defined there and kept apart from B = 64, the number of intensity bins used for conditioning.
  5. p5 l240, (b_L, b_R). These switches existed only for the ablation. They are removed from the equations, and the ablation variants are described in words in Sec. 4.3.
  6. p5 l264, tile rasterizer. It is the tile-based rasterizer of the N-DGS implementation, which follows the 3DGS design of Kerbl et al. Sec. 3.4 now names it.
  7. p6 l304, r = 8. Sec. 4.1 now says that r is the size of the response code of the local TF encoder, which has a 64-unit hidden layer.
  8. p6 l308, why 0.04. 0.04 is about 10 of 255 intensity levels. It selects the pixels that the edit visibly changes in the reference image, and it is used only for evaluation. The conclusions do not depend on it: the heart and knee comparison is the same at 0.01, 0.02, 0.04 and 0.08, and with all pixels and no threshold.
  9. p7 l342, Table 2. See W2: counts for each component, and Figure 5.
  10. p8 l429, Fig. 3. The new Figure 5 shows zoomed crops at identical positions in every column, and Figure 6 compares with Render-FM with edit-error maps.
  11. p9 l435–441. Figure 5 includes the preset-embedding and gate-off variants next to the full model, with edit-error maps.
  12. p13 l650, appearance scales. They are fixed design constants, the same for all scans, and were not tuned per scan or on held-out TFs. The lookup colour scale is 1, so the authored colour change is applied as written. Both opacity scales are 4 because the offsets act on the opacity logit, whose slope is 4 at an opacity of 0.5. The residual colour scale of 0.25 bounds the learned colour correction, so that it stays a correction to the lookup. Learned gains can rescale the lookup during training. App. A now gives this rationale.
Reviewer_ubDhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FactorSplat is a per-scene Gaussian proxy for medical volume rendering and this technique accepts new transfer functions at inference time. FactorSplat shares Gaussian centers, covariances, and the view-dependent N-DGS opacity gate across presets. Each primitive keeps a fixed set of local region and intensity samples. The paper’s contribution: conditioning one N-DGS proxy on region-specific intensity-to-RGBA curves, lookup/residual appearance model with coverage-aware pruning. The paper also discusses the comparison to a region-aware version of VEG.

Critical strengths

The paper reframes the transfer function (TF) as an inference-time functional control over a Gaussian proxy. This seems to be better than the original methods like using a categorical preset or a per-preset model. An essential feature is that the geometry is provably invariant to TF edits. The model size grows with the number of Gaussians, not with the number of TFs. This goes meaningfully beyond region-aware VEG, iVR-GS, and ECoNGS.

The evaluation spans seven scans over 40+ conditions for each. This actually tests whether the conditioning generalizes, rather than just interpolating between presets.

Critical weaknesses

One weakness is that, the evaluation does not convincingly isolate the contribution. The comparison over VEG is an essential evaluation of the paper, but the comparison involves a custom VEG extension. And FS performs worse than VEG on edit fidelity for the heart and knee-joint scans.

The method also assumes a correct segmentation and then freezes it. Each Gaussian keeps the tissue label from its initial position. During training the Gaussian moves, while that label stays put. A wrong mask, or a Gaussian that drifts into another tissue, makes the color lookup use the wrong curve.

Questions for authors

On the same held-out views, what is the PSNR of the cached fast renderer, and what is the frame rate of the quality renderer? Appendix E says the two renderers can produce different images. Table 1 puts the quality renderer’s PSNR next to the fast renderer’s speed. If the fast renderer stays close to the PSNR in Table 1, the reported speed applies to the reported image quality. If it falls short, the speed describes a different image.

On the heart scan, the changed-region error is worse than VEG on every split, including where PSNR is higher. On the knee scan, VEG wins that edit error on every split. Is this a side effect of the 0.04 threshold, or do the edited regions actually miss the reference? If the edit gap remains, the 1.1–1.5 dB average is too coarse to support the claim.

2 comments
AuthorsOpenReview ↗

Response to Reviewer ubDh

Thank you for the careful review. Both weaknesses and both questions are answered with new measurements. The higher edit error on heart is not a threshold effect: a control that re-samples the fixed tissue samples puts FactorSplat ahead of VEG in all four splits. The reported speed applies to the reported quality: the cached fast renderer loses 0.004 dB on the same views.

All numbers refer to the revised PDF. Tables 3–5 of the submission are now Tables 4–6, and the new Table 3 is the paired renderer comparison.

W1. The comparison uses our extension of VEG and does not isolate the contribution

The extension changes how VEG reads the TF and nothing else, and the contribution is isolated by ablations inside FactorSplat. VEG takes one TF for the whole volume, while our scenes use one curve per tissue region. Each VEG Gaussian is assigned a region and reads that region's curve. Initialization, the 100-bin lookup, shading, optimizer, losses and density control are those of the released code, and VEG is trained on the same 24 TFs and cameras as FactorSplat. The only other change is a fix for a non-finite gradient in the specular term, without which training stops (App. B).

The comparison with VEG is a comparison of two complete methods under one protocol. The contribution of each component is measured inside FactorSplat, where one component is removed at a time (Tables 2 and 5; heart and vascular, 4 splits each):

Removed componentPSNR lower inLossEdit error higher in
Learned residual8 of 80.50–1.60 dB8 of 8
TF lookup8 of 80.89–3.47 dB8 of 8

Table 1 also gives two references that do not depend on VEG: an unconditioned proxy, which ignores the edit, and one specialist model fitted per TF.

W2. The method assumes a correct segmentation and keeps it fixed

The segmentation defines the target appearance; FactorSplat does not need it to be anatomically correct to fit that target. The segmentation mask is used only to define the TF: it says which curve applies to which voxels. The reference renderer uses the same mask to produce the target images. A wrong label therefore gives that region an undesired TF colour in the reference image itself, and FactorSplat reproduces that image like any other. The error is in the authored appearance, which is the target the proxy is trained to reproduce.

Keeping the tissue samples fixed is a design choice that can be relaxed: re-sampling them during training puts FactorSplat ahead of VEG on heart in all four splits, and improves knee in all four. The only assumption FactorSplat adds is that a Gaussian stays in the tissue where it started. A Gaussian does not carry a single label. It stores up to 27 weighted (tissue, intensity) samples from the voxels around its initial position, so a Gaussian on a boundary mixes the curves of the tissues it covers. In a control run we re-sample them from the current geometry after each densification step (Table 8).

Heart, validation / interpolation / composition / OOD:

PSNR (dB)Edit error ×100
FactorSplat, fixed samples30.76 / 29.32 / 28.63 / 23.701.32 / 2.62 / 3.17 / 9.78
FactorSplat, re-sampled31.12 / 30.34 / 30.15 / 26.821.09 / 2.03 / 2.36 / 5.87
VEG (Table 4)28.79 / 28.21 / 27.80 / 25.701.22 / 2.26 / 3.03 / 5.93

On heart, re-sampling removes the gap to VEG in both metrics. On knee it improves both metrics in all four splits and brings the edit error below VEG on interpolation and OOD (Table 8).

On hand, where FactorSplat already leads VEG, re-sampling is neutral on three splits and costs 0.58 dB on OOD. We therefore report re-sampling as a control (one seed per run) and keep the submitted method and its results unchanged.

AuthorsOpenReview ↗

Response to Reviewer ubDh

Q1. PSNR of the cached fast renderer and FPS of the quality renderer on the same views

The fast renderer stays at the PSNR of the quality renderer (it loses 0.004 dB) and is 2.8× faster. Both are measured on the same 56 views (8 validation views for each of the seven scans, 1600 × 1600; Table 3). The names say how a path renders, not how good its image is: "quality" recomputes the appearance every frame with the rasterizer used in training, "cached" computes it once per TF, and "fast" adds the speed-optimized rasterizer.

RendererPSNR (dB)FPS, B200FPS, A100
Training rasterizer ("quality")35.20201.6134.9
+ cached appearance ("cached quality")35.20229.5150.1
+ fast rasterizer ("cached fast")35.20557.6522.1

Over all 4,760 held-out images the mean loss is 0.0008 dB (32.7545 to 32.7537). The 524 FPS and 1.17 ms of the submission are unchanged (Table 1). Table 3 is a separate run that pairs quality and speed, and it measures 557.6 FPS and 1.10 ms on B200.

Q2. Is the higher edit error on heart and knee a side effect of the 0.04 threshold?

No. The threshold does not change the comparison, and the re-sampling control identifies the fixed tissue samples as a contributing factor: relaxing them removes the gap on heart (W2 above). The comparison on these two scans does not change with the threshold: it is the same in 31 of 32 scan–split cells at thresholds 0.01, 0.02, 0.04 and 0.08 and in all 8 cells with no threshold. With re-sampled tissue samples, FactorSplat has the lower edit error on heart in all four splits, and on knee in interpolation and OOD (Table 8).

Heart and knee are the exception and not the rule. On the other five scans FactorSplat has the lower edit error in 19 of 20 scan–split cells. Over all seven scans it has the higher PSNR in 24 of 28 cells, including three of the four on heart. Sec. 4.2 now states the claim with these counts and names the two scans.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the three reviewers for their careful reading. A revised PDF is uploaded, and all table, figure and section numbers in our replies refer to it. The method and the results of the submission are unchanged. The revision adds the measurements the reviewers asked for and a direct comparison with Render-FM.

  1. What is new relative to Render-FM (2vdj). Render-FM takes the TF as input and predicts all Gaussians again for every TF, which takes 2.6–2.8 s. FactorSplat keeps one proxy per scan and updates only its colour and opacity in 1.17 ms, with the geometry unchanged, at native resolution and on both CT and MR. The revised text describes Render-FM in these terms. Against the pretrained Render-FM model, FactorSplat has less than half the edit error on both tested scans (Table 11, Fig. 6).

  2. The reported speed applies to the reported quality (ubDh, cQ2G, 2vdj). On the same views, the cached fast renderer loses 0.004 dB to the quality renderer, and 0.0008 dB over all 4,760 held-out images (Table 3). The 524 FPS and 1.17 ms of the submission are unchanged; the paired run measured 557.6 FPS on B200 and 522.1 FPS on A100.

  3. FactorSplat is ahead of VEG in 24 of 28 PSNR comparisons and 19 of 28 edit-error comparisons (ubDh). The exceptions are the edit error on heart and knee, which Sec. 4.2 now names. They are not an effect of the 0.04 threshold. A control that re-computes the tissue samples during training removes the gap on heart and narrows it on knee (Table 8).

  4. The reference path tracer does not reach this speed at this quality (cQ2G, 2vdj). On the same views and GPU, one path-tracer call is 25 times slower than FactorSplat and 1.71 dB less accurate. At the tested budget of 16 the path tracer is the more accurate, at 268 ms per frame, about 140 times FactorSplat's frame time (Table 10).

  5. New controls (cQ2G). An ablation of coverage-aware pruning (Table 8), the initialization of every scan (Table 7), and storage against the voxel volumes (Table 9).

  6. Presentation (2vdj). Sec. 3 explains each step before its equation, the twelve line comments are addressed, Figure 5 adds zoomed crops for the ablations, and Figure 6 a visual comparison with Render-FM.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.