- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduced a 3D Gaussian VAE encodes Gaussian secenes into mult-slot tripline latents. A point-transformer and cell-wise queries keeps the information from the overlapping projections. A set deocder use optimal matching with rendering supervision to reconstruct adaptive local Gaussian output set. Finally, a flow model generates scene from full point cloud. It also support generation from single image with known camera information.
The method is evaluated on VAE reconstruction, conditional generation and completion tasks.
Critical strengths
The architechure design address the irregular-set and reglar-latent mismatch. Using multiple encoder slots to reduce projection colllisions, and use local set and phantom voxels to address the variable primitive counts. The design is also supported by the ablation study.
The results are good enough compared to the baseline. The PSNR and LPIPS improvement are substantial, this also validated in the visual results.
Critical weaknesses
The scope of 3D Gaussian VAE is a bit limited, the authors do provide some interesting applications, but it is unclear the benifit of having such a VAE. I can think of better transfer speed by reducing the number of Gaussians?
The conditional generation might need to compare with other baseline methods. I am also not quite get the idea behind the point cloud based generation. Is this a point cloud based reconstruction?
The image based generation quality is hard to judge. The visible part looks fine, but the invisible region are far from ideal, for an image based generation, people expect good quality, or at least reasonable generation / completeion over unseen regions.
Can3Tok is limited to 40,000 input primitives despite source crops containing hundreds of thousands, and both Can3Tok and DiffGS collapse to low-detail solutions under the adopted configuration.
Questions for authors
I am a bit confused about the conditional-generation part though. The paper seems also want to emphasis its contribution on conditional generation? Is this an application? or the method is designed for such task?
If the author want to hint that the VAE can be used for Gaussian Flow generation, then it need to be implemented in a full functioned generation and compared with other methods. The current application will only work as a proof-of-concept, but still a long way to be an actual generation.
Some details should be better discussed like latent sizes, model sizes, training procedure etc.
6 comments
Response to Reviewer YA1A (Part 2/4)
W2 & Q2: W2. The conditional generation might need to compare with other baselines methods. I am also not quite get the idea behind the point cloud based generation. Is this a point cloud based reconstruction? Q2. If the author want to hint that the VAE can be used for Gaussian Flow generation, then it need to be implemented in a full functioned generation and compared with other methods. The current application will only work as a proof-of-concept, but still a long way to be an actual generation.
To W2 & Q2: Conditional generation comparisons and point-cloud conditioning
We thank the reviewer for this helpful question and suggestion.
1. Motivation behind point-cloud-conditioned generation.
Is the point-cloud setting reconstruction? Yes. Given a complete colored point cloud, this experiment reconstructs a native Gaussian scene by synthesizing the missing scale, rotation, and opacity attributes. This setting provides a reference with complete geometric conditioning to complement our partial-view amodal generation. Our flow model takes complete geometry and color priors and conditionally synthesizes the missing native 3DGS attributes. This allows us to examine attribute synthesis in the learned triplane latent space when geometric information is provided. We have clarified this motivation in Sec. 4.3 of the revised manuscript.
2. Quantitative and qualitative comparison on single-image generation.
Is this a full generation system? The generators are complete, trained systems: a 1.1B-parameter flow transformer well trained (Table 16), evaluated on held-out InteriorGS scenes against an image-conditioned GaussianGPT AR baseline with the same conditioning images, split, and metrics. Appendix A.4 details its tokenizer, conditioning mechanism, and training and sampling settings. We excluded Can3Tok and DiffGS from the generation comparison because their autoencoders produced coarse scene reconstructions (Table 1, Fig. 10), making it difficult to assess the quality of a generator trained on their latents.
Table R2: Quantitative comparison of single-image-conditioned scene generation on InteriorGS. Both methods are evaluated under the identical split, conditioning views, and 3D depth-reprojection masks. KID and MMD-CD are ×10³; COV-CD is in %.
| Method | Obs. FID ↓ | Obs. KID ↓ | Unobs. FID ↓ | Unobs. KID ↓ | Unobs. COV-CD ↑ | Unobs. MMD-CD ↓ |
|---|---|---|---|---|---|---|
| GaussianGPT-AR | 84.67 | 37.85 | 92.69 | 43.02 | 64.30 | 31.54 |
| TriFold (Ours) | 60.38 | 18.20 | 67.81 | 21.43 | 85.24 | 12.75 |
| Δ (TriFold − GaussianGPT-AR) | −24.29 | −19.65 | −24.88 | −21.59 | +20.94 | −18.79 |
As shown in Table R2, TriFold outperforms GaussianGPT-AR across all observed and unobserved metrics, lowering unobserved FID by 24.88 points (67.81 vs. 92.69) and reducing unobserved geometric error (MMD-CD) by nearly 60% (12.75 vs. 31.54). We have incorporated these quantitative results into Table 2 and added side-by-side qualitative generation comparisons in Fig. 5 (and Fig. 15 in Appendix A.11) of the revised manuscript.
W3: The image based generation quality is hard to judge. The visible part looks fine, but the invisible region are far from ideal, for an image based generation, people expect good quality, or at least reasonable generation / completeion over unseen regions.
To W3: Generation quality and evaluation in unobserved regions
Thank you for pointing this out. To avoid judging unseen areas by visual inspection alone, we have explicitly decoupled and evaluated unobserved regions using 3D depth-reprojection masks (Appendix A.6):
Quantitative evaluation of unobserved regions. Rather than relying solely on visual inspection, Table 2 explicitly decouples observed and unobserved regions using 3D depth-reprojection masks (Appendix A.6). Specifically, valid pixels from the input view are lifted to 3D and reprojected into novel evaluation views to mask out visible areas. Within these regions marked as unobserved by our visibility masks, we evaluate both distributional appearance (FID / KID) and geometric coverage and fidelity (COV-CD / MMD-CD) against ground truth. TriFold achieves 67.81 FID, 21.43 × 10⁻³ KID, 85.24% COV-CD, and 12.75 × 10⁻³ MMD-CD, substantially outperforming GaussianGPT under the exact same protocol (as detailed in the To W2&Q2 section Table R2).
Visual smoothness in unseen regions. We agree that occluded areas appear smoother than visible regions. This may partly reflect the ambiguity of inferring hidden layouts from a single view, where multiple completions can be plausible. Increasing the diversity of 3DGS training scenes and the generator capacity remains a promising direction for improving occluded details.
Response to Reviewer YA1A (Part 3/4)
W4: Can3Tok is limited to 40,000 input primitives despite source crops containing hundreds of thousands, and both Can3Tok and DiffGS collapse to low-detail solutions under the adopted configuration.
To W4: Baselines capacity limits and low-detail convergence at scene scale
Thank you for this observation. Both baselines were trained from scratch using their official implementations and method-specific optimization settings, with the scene-level adaptations documented in Appendix A.4. Under these configurations, both produce low-detail reconstructions. For Can3Tok, this limitation persists under the windowed evaluation protocol, which avoids global input downsampling.
Can3Tok: We partitioned each evaluation block into spatially coherent 40K-Gaussian windows, reconstructed each window independently, and merged the outputs into a full-block reconstruction. As shown in Figure 8, these partitioned 40K ground-truth windows retain recognizable local structures and appearance details prior to encoding, whereas Can3Tok's corresponding reconstructions collapse into coarse envelopes without preserving scene-specific details.
DiffGS: Trained on the exact same normalized crops, DiffGS interpolates Gaussians from continuous spatial fields, whose smoothness bias may average out sharp local variations in scale, rotation, and appearance.
Training convergence. We also report the PSNR convergence of Can3Tok and DiffGS in Table R6 of Reviewer MRST Q2, showing that both methods largely converge by 80K training steps, with only marginal improvements from 60K to 80K (+0.05 dB and +0.15 dB, respectively).
Response to Reviewer YA1A (Part 1/4)
We sincerely thank the reviewer for the constructive questions about the purpose of the Gaussian VAE, the role of conditional generation, and the experimental evidence. In response, we have 1) clarified the representation-learning contribution through cross-architecture and cross-representation experiments, 2) clarified point-cloud-conditioned synthesis and added quantitative and qualitative generation comparisons with a GaussianGPT-based image-conditioned AR model, 3) expanded the evaluation of generated results and input-view consistency, and 4) clarified baseline adaptations and consolidated latent sizes, model sizes, and training procedures. We have also redesigned Figure 2 to make the architecture easier to follow. We address each concern and present the supporting results below.
W1& Q1: W1. The scope of 3D Gaussian VAE is a bit limited, the authors do provide some interesting applications, but it is unclear the benifit of having such a VAE. I can think of better transfer speed by reducing the number of Gaussians? Q1. I am a bit confused about the conditional-generation part though. The paper seems also want to emphasis its contribution on conditional generation? Is this an application? or the method is designed for such task?
To W1&Q1: Purpose and methodological contribution
Thank you for raising this question. Our primary goal is to enable high-fidelity generative modeling and content-adaptive reconstruction of native 3D Gaussian scenes by folding irregular primitives into structured latent representations, rather than optimizing data transfer speed via Gaussian pruning. Because native 3DGS consists of unordered, variable-cardinality primitives with nonuniform spatial densities, it is difficult to directly model with standard generative backbones that operate on regular tensor formats. TriFold bridges this gap by learning structured latents that jointly preserve geometry and appearance without imposing a rigid scene-level budget.
Our core contribution lies in this representation-learning paradigm, with our triplane VAE serving as its primary, generative-friendly instantiation:
Encoding irregular primitives (Sec. 3.2): Our encoder maps non-uniform Gaussian sets into structured latents while preserving local attribute heterogeneity through cell-wise query aggregation and sparse cross-plane attention.
Content-adaptive decoding (Sec. 3.3): Our decoder reconstructs variable-cardinality Gaussian scenes through occupancy-guided support proposal and OT-guided set decoding.
To demonstrate that our methodological effectiveness stems from these representation-learning designs rather than simply adopting a specific latent format, we conducted cross-architecture and cross-representation evaluations as shown in Table R4:
Table R4: Cross-architecture and cross-representation ablation across latent topologies and bottlenecks. All variants are trained from scratch under the identical combined InteriorGS and Aria 3DGS training protocol and evaluated on the same InteriorGS test set used for Table 1.
These results highlight two critical insights regarding our methodology:
The power of our representation-learning framework (Row 0 vs. Row 1): Replacing GaussianGPT's encoder–decoder with TriFold's while retaining its exact sparse voxel grid and VQ bottleneck substantially improves PSNR from 21.40 to 24.36 dB (+2.96 dB), demonstrating that our folding and set-decoding paradigm generalizes across latent structures.
Encoder–decoder design and latent layout (Rows 0–4): While pairing our framework with GaussianGPT's sparse sequence achieves 24.36 dB (Row 1), combining it with our 2D triplane representation (Row 2) yields the highest discrete fidelity (24.58 dB) while providing a regular 2D spatial layout for downstream generation. Both TriFold and the adapted DiffGS baseline use continuous triplane latents, with reconstruction PSNRs of 26.55 dB (Row 3) and 12.43 dB (Row 4), respectively.
We have refined our contribution in Sec.1 and included these cross-architecture analyses in the Appendix.
In the revised manuscript:
Lines 110–113: Native scene-level 3D Gaussian representation learning. We introduce TriFold, a variational autoencoder that learns structured representations of Gaussian scenes with variable cardinality and nonuniform spatial density by folding irregular primitives into fixed-layout latents while supporting content-adaptive local Gaussian allocation during decoding.