acceptodds
Under review as a conference paper at ICLR 2027

TriFold: Folding irregular 3D Gaussians into structured latents for 3D Scene Generation

Abstract

Recent progress in 3D generation has motivated growing interest in compact generative representations of complex 3D scenes. For native 3D Gaussian scenes, however, learning such representations remains challenging due to widely varying primitive counts, nonuniform spatial density, and differences in geometry and appearance even among spatially neighboring primitives. In this work, we introduce TriFold, a native 3D Gaussian VAE that encodes Gaussian scenes into a fixed-layout triplane latent and jointly reconstructs their geometry and appearance. The key insight is to impose structural regularity on the latent space while preserving the spatial adaptivity of the underlying Gaussians. During encoding, we use cell-wise learned queries to aggregate projected Gaussians into multiple slots per cell, reducing the information loss caused by single-feature pooling. During decoding, occupancy prediction identifies active regions, where query-based set prediction and candidate selection reconstruct local Gaussian sets with spatially varying primitive counts. Building on this representation, we train a latent flow-matching model for conditional scene-level 3DGS generation, enabling point-cloud-conditioned synthesis and single-image-conditioned amodal generation. Experiments demonstrate that TriFold achieves high-fidelity reconstruction and generation from a single image that provides only a partial observation of the scene, validating its effectiveness for native 3D Gaussian representation learning.

Primary area:
generative models
Keywords:
3D generation & Scene generation & 3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_YA1AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduced a 3D Gaussian VAE encodes Gaussian secenes into mult-slot tripline latents. A point-transformer and cell-wise queries keeps the information from the overlapping projections. A set deocder use optimal matching with rendering supervision to reconstruct adaptive local Gaussian output set. Finally, a flow model generates scene from full point cloud. It also support generation from single image with known camera information.

The method is evaluated on VAE reconstruction, conditional generation and completion tasks.

Critical strengths

The architechure design address the irregular-set and reglar-latent mismatch. Using multiple encoder slots to reduce projection colllisions, and use local set and phantom voxels to address the variable primitive counts. The design is also supported by the ablation study.

The results are good enough compared to the baseline. The PSNR and LPIPS improvement are substantial, this also validated in the visual results.

Critical weaknesses

The scope of 3D Gaussian VAE is a bit limited, the authors do provide some interesting applications, but it is unclear the benifit of having such a VAE. I can think of better transfer speed by reducing the number of Gaussians?

The conditional generation might need to compare with other baseline methods. I am also not quite get the idea behind the point cloud based generation. Is this a point cloud based reconstruction?

The image based generation quality is hard to judge. The visible part looks fine, but the invisible region are far from ideal, for an image based generation, people expect good quality, or at least reasonable generation / completeion over unseen regions.

Can3Tok is limited to 40,000 input primitives despite source crops containing hundreds of thousands, and both Can3Tok and DiffGS collapse to low-detail solutions under the adopted configuration.

Questions for authors

I am a bit confused about the conditional-generation part though. The paper seems also want to emphasis its contribution on conditional generation? Is this an application? or the method is designed for such task?

If the author want to hint that the VAE can be used for Gaussian Flow generation, then it need to be implemented in a full functioned generation and compared with other methods. The current application will only work as a proof-of-concept, but still a long way to be an actual generation.

Some details should be better discussed like latent sizes, model sizes, training procedure etc.

6 comments
AuthorsOpenReview ↗

Response to Reviewer YA1A (Part 1/4)

We sincerely thank the reviewer for the constructive questions about the purpose of the Gaussian VAE, the role of conditional generation, and the experimental evidence. In response, we have 1) clarified the representation-learning contribution through cross-architecture and cross-representation experiments, 2) clarified point-cloud-conditioned synthesis and added quantitative and qualitative generation comparisons with a GaussianGPT-based image-conditioned AR model, 3) expanded the evaluation of generated results and input-view consistency, and 4) clarified baseline adaptations and consolidated latent sizes, model sizes, and training procedures. We have also redesigned Figure 2 to make the architecture easier to follow. We address each concern and present the supporting results below.

W1& Q1: W1. The scope of 3D Gaussian VAE is a bit limited, the authors do provide some interesting applications, but it is unclear the benifit of having such a VAE. I can think of better transfer speed by reducing the number of Gaussians? Q1. I am a bit confused about the conditional-generation part though. The paper seems also want to emphasis its contribution on conditional generation? Is this an application? or the method is designed for such task?

To W1&Q1: Purpose and methodological contribution

Thank you for raising this question. Our primary goal is to enable high-fidelity generative modeling and content-adaptive reconstruction of native 3D Gaussian scenes by folding irregular primitives into structured latent representations, rather than optimizing data transfer speed via Gaussian pruning. Because native 3DGS consists of unordered, variable-cardinality primitives with nonuniform spatial densities, it is difficult to directly model with standard generative backbones that operate on regular tensor formats. TriFold bridges this gap by learning structured latents that jointly preserve geometry and appearance without imposing a rigid scene-level budget.

Our core contribution lies in this representation-learning paradigm, with our triplane VAE serving as its primary, generative-friendly instantiation:

Encoding irregular primitives (Sec. 3.2): Our encoder maps non-uniform Gaussian sets into structured latents while preserving local attribute heterogeneity through cell-wise query aggregation and sparse cross-plane attention.

Content-adaptive decoding (Sec. 3.3): Our decoder reconstructs variable-cardinality Gaussian scenes through occupancy-guided support proposal and OT-guided set decoding.

To demonstrate that our methodological effectiveness stems from these representation-learning designs rather than simply adopting a specific latent format, we conducted cross-architecture and cross-representation evaluations as shown in Table R4:

Table R4: Cross-architecture and cross-representation ablation across latent topologies and bottlenecks. All variants are trained from scratch under the identical combined InteriorGS and Aria 3DGS training protocol and evaluated on the same InteriorGS test set used for Table 1.

IDEncoder / DecoderLatent RepresentationVAE BottleneckPSNR ↑SSIM ↑LPIPS ↓
0GaussianGPTSparse voxel gridVQ-VAE21.400.8560.157
1TriFoldSparse voxel gridVQ-VAE24.360.8960.102
2TriFoldTriplanesVQ-VAE24.580.9010.097
3TriFold (Ours)TriplanesContinuous VAE26.550.9130.088
4DiffGSTriplanesContinuous VAE12.430.7190.374

These results highlight two critical insights regarding our methodology:

The power of our representation-learning framework (Row 0 vs. Row 1): Replacing GaussianGPT's encoder–decoder with TriFold's while retaining its exact sparse voxel grid and VQ bottleneck substantially improves PSNR from 21.40 to 24.36 dB (+2.96 dB), demonstrating that our folding and set-decoding paradigm generalizes across latent structures.

Encoder–decoder design and latent layout (Rows 0–4): While pairing our framework with GaussianGPT's sparse sequence achieves 24.36 dB (Row 1), combining it with our 2D triplane representation (Row 2) yields the highest discrete fidelity (24.58 dB) while providing a regular 2D spatial layout for downstream generation. Both TriFold and the adapted DiffGS baseline use continuous triplane latents, with reconstruction PSNRs of 26.55 dB (Row 3) and 12.43 dB (Row 4), respectively.

We have refined our contribution in Sec.1 and included these cross-architecture analyses in the Appendix.

In the revised manuscript:

Lines 110–113: Native scene-level 3D Gaussian representation learning. We introduce TriFold, a variational autoencoder that learns structured representations of Gaussian scenes with variable cardinality and nonuniform spatial density by folding irregular primitives into fixed-layout latents while supporting content-adaptive local Gaussian allocation during decoding.

AuthorsOpenReview ↗

Response to Reviewer YA1A (Part 2/4)

W2 & Q2: W2. The conditional generation might need to compare with other baselines methods. I am also not quite get the idea behind the point cloud based generation. Is this a point cloud based reconstruction? Q2. If the author want to hint that the VAE can be used for Gaussian Flow generation, then it need to be implemented in a full functioned generation and compared with other methods. The current application will only work as a proof-of-concept, but still a long way to be an actual generation.

To W2 & Q2: Conditional generation comparisons and point-cloud conditioning

We thank the reviewer for this helpful question and suggestion.

1. Motivation behind point-cloud-conditioned generation.

Is the point-cloud setting reconstruction? Yes. Given a complete colored point cloud, this experiment reconstructs a native Gaussian scene by synthesizing the missing scale, rotation, and opacity attributes. This setting provides a reference with complete geometric conditioning to complement our partial-view amodal generation. Our flow model takes complete geometry and color priors and conditionally synthesizes the missing native 3DGS attributes. This allows us to examine attribute synthesis in the learned triplane latent space when geometric information is provided. We have clarified this motivation in Sec. 4.3 of the revised manuscript.

2. Quantitative and qualitative comparison on single-image generation.

Is this a full generation system? The generators are complete, trained systems: a 1.1B-parameter flow transformer well trained (Table 16), evaluated on held-out InteriorGS scenes against an image-conditioned GaussianGPT AR baseline with the same conditioning images, split, and metrics. Appendix A.4 details its tokenizer, conditioning mechanism, and training and sampling settings. We excluded Can3Tok and DiffGS from the generation comparison because their autoencoders produced coarse scene reconstructions (Table 1, Fig. 10), making it difficult to assess the quality of a generator trained on their latents.

Table R2: Quantitative comparison of single-image-conditioned scene generation on InteriorGS. Both methods are evaluated under the identical split, conditioning views, and 3D depth-reprojection masks. KID and MMD-CD are ×10³; COV-CD is in %.

MethodObs. FID ↓Obs. KID ↓Unobs. FID ↓Unobs. KID ↓Unobs. COV-CD ↑Unobs. MMD-CD ↓
GaussianGPT-AR84.6737.8592.6943.0264.3031.54
TriFold (Ours)60.3818.2067.8121.4385.2412.75
Δ (TriFold − GaussianGPT-AR)−24.29−19.65−24.88−21.59+20.94−18.79

As shown in Table R2, TriFold outperforms GaussianGPT-AR across all observed and unobserved metrics, lowering unobserved FID by 24.88 points (67.81 vs. 92.69) and reducing unobserved geometric error (MMD-CD) by nearly 60% (12.75 vs. 31.54). We have incorporated these quantitative results into Table 2 and added side-by-side qualitative generation comparisons in Fig. 5 (and Fig. 15 in Appendix A.11) of the revised manuscript.

W3: The image based generation quality is hard to judge. The visible part looks fine, but the invisible region are far from ideal, for an image based generation, people expect good quality, or at least reasonable generation / completeion over unseen regions.

To W3: Generation quality and evaluation in unobserved regions

Thank you for pointing this out. To avoid judging unseen areas by visual inspection alone, we have explicitly decoupled and evaluated unobserved regions using 3D depth-reprojection masks (Appendix A.6):

Quantitative evaluation of unobserved regions. Rather than relying solely on visual inspection, Table 2 explicitly decouples observed and unobserved regions using 3D depth-reprojection masks (Appendix A.6). Specifically, valid pixels from the input view are lifted to 3D and reprojected into novel evaluation views to mask out visible areas. Within these regions marked as unobserved by our visibility masks, we evaluate both distributional appearance (FID / KID) and geometric coverage and fidelity (COV-CD / MMD-CD) against ground truth. TriFold achieves 67.81 FID, 21.43 × 10⁻³ KID, 85.24% COV-CD, and 12.75 × 10⁻³ MMD-CD, substantially outperforming GaussianGPT under the exact same protocol (as detailed in the To W2&Q2 section Table R2).

Visual smoothness in unseen regions. We agree that occluded areas appear smoother than visible regions. This may partly reflect the ambiguity of inferring hidden layouts from a single view, where multiple completions can be plausible. Increasing the diversity of 3DGS training scenes and the generator capacity remains a promising direction for improving occluded details.

AuthorsOpenReview ↗

Response to Reviewer YA1A (Part 3/4)

W4: Can3Tok is limited to 40,000 input primitives despite source crops containing hundreds of thousands, and both Can3Tok and DiffGS collapse to low-detail solutions under the adopted configuration.

To W4: Baselines capacity limits and low-detail convergence at scene scale

Thank you for this observation. Both baselines were trained from scratch using their official implementations and method-specific optimization settings, with the scene-level adaptations documented in Appendix A.4. Under these configurations, both produce low-detail reconstructions. For Can3Tok, this limitation persists under the windowed evaluation protocol, which avoids global input downsampling.

Can3Tok: We partitioned each evaluation block into spatially coherent 40K-Gaussian windows, reconstructed each window independently, and merged the outputs into a full-block reconstruction. As shown in Figure 8, these partitioned 40K ground-truth windows retain recognizable local structures and appearance details prior to encoding, whereas Can3Tok's corresponding reconstructions collapse into coarse envelopes without preserving scene-specific details.

DiffGS: Trained on the exact same 2563256^3 normalized crops, DiffGS interpolates Gaussians from continuous spatial fields, whose smoothness bias may average out sharp local variations in scale, rotation, and appearance.

Training convergence. We also report the PSNR convergence of Can3Tok and DiffGS in Table R6 of Reviewer MRST Q2, showing that both methods largely converge by 80K training steps, with only marginal improvements from 60K to 80K (+0.05 dB and +0.15 dB, respectively).

Reviewer_MRSTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TriFold, a method that encodes 3D Gaussians with varying counts and spatial densities into a fixed-size triplane latent. Multiple feature slots store information from Gaussians projected onto the same plane cell. Occupancy prediction and a set decoder then reconstruct a variable number of Gaussians in each region. The paper evaluates reconstruction on InteriorGS and Aria and demonstrates image-conditioned and point-cloud-conditioned scene generation using the learned latents.

Critical strengths
  • The design addresses the varying numbers and spatial densities of Gaussians across scenes. The encoder uses multiple feature slots, while the decoder adjusts the Gaussian count in each region. Together, these components allow a fixed-size latent to represent irregular Gaussian sets.

  • The reconstruction gains are substantial. Table 1 reports a 5.15 dB PSNR improvement over GaussianGPT, with strong results on the other metrics as well. Ablations on the encoder, decoder, supervision, and representation capacity support the proposed design choices.

Critical weaknesses
  • Comparisons with existing generative methods are missing. Table 2 only compares two conditioning settings of TriFold, making it difficult to assess whether the learned latent offers an advantage over existing representations for scene generation. FID and KID also do not directly measure whether individual outputs preserve the content and colors of their conditioning images. An evaluation of this consistency would strengthen the generation results.

  • The poor reconstruction results of some baselines in Table 1 need further explanation. Appendix A.4 and Figure 9 show that Can3Tok and DiffGS produce similar low-quality scenes from different inputs, raising questions about their training and convergence under the current setup. Can3Tok also requires reducing each input scene to 40,000 Gaussians. The selection procedure and the rendering quality after this reduction should be clarified. Without this analysis, it is difficult to distinguish limitations of the representations from errors introduced by input reduction or difficulties adapting the methods to the current data.

Questions for authors
  • What specific differences in conditioning prevent direct comparison with existing scene generation methods in Table 2? Please share any existing comparisons or evaluations of consistency between the conditioning images and generated outputs.

  • What training and convergence checks were performed for Can3Tok and DiffGS? For Can3Tok, how were the 40,000 Gaussians selected from each pre-fitted scene? If available, please provide renderings or quality measurements after this reduction and before encoding. These would help separate errors introduced by input reduction from those introduced by the reconstruction model.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MRST (Part 1/2)

We sincerely thank the reviewer for the constructive feedback and for recognizing TriFold’s adaptive representation design and substantial reconstruction gains. In response to your comments, we have clarified the baseline training and input-reduction protocols, and strengthened our evaluation with: 1) a single-image conditional generation comparison against a scene-level GaussianGPT AR model, 2) input-view rendering metrics (PSNR, SSIM, and LPIPS) to directly evaluate conditioning consistency, and 3) convergence diagnostics for Can3Tok and DiffGS and input-selection analyses for Can3Tok. We address each point in detail below.

W1&Q1: W1. Comparisons with existing generative methods are missing. Table 2 only compares two conditioning settings of TriFold, making it difficult to assess whether the learned latent offers an advantage over existing representations for scene generation. FID and KID also do not directly measure whether individual outputs preserve the content and colors of their conditioning images. An evaluation of this consistency would strengthen the generation results. Q1. What specific differences in conditioning prevent direct comparison with existing scene generation methods in Table 2? Please share any existing comparisons or evaluations of consistency between the conditioning images and generated outputs.

To W1&Q1: Generation comparison and conditioning consistency

We thank the reviewer for highlighting the need for a generative baseline and a direct metric for conditioning consistency.

Existing 3DGS generative methods do not directly match our single-image amodal scene completion setup: GaussianGPT’s published completion task starts from a partial 3D Gaussian scene rather than a single RGB image, while Can3Tok and DiffGS converge to coarse envelopes at scene scale (Table 1, Fig. 10). We selected GaussianGPT for the generation comparison because it achieved the highest reconstruction PSNR among the baselines in Table 1 (21.40 dB). We adapted its autoregressive model for single-image conditioning and evaluated it alongside TriFold using the same InteriorGS split, conditioning images, and evaluation views. Appendix A.4 further describes the implementation details.

Table 2 reports appearance quality in observed and unobserved regions and geometry quality in unobserved regions. To assess consistency with the conditioning image for each individual sample, we also render the generated scene from the input camera and compute PSNR, SSIM, and LPIPS against the original RGB image.

Table R5: Quantitative comparison of single-image-conditioned scene generation and input-view consistency on InteriorGS. Both methods are evaluated under the identical split, conditioning views, and 3D depth-reprojection masks. Input-view metrics measure rendering consistency against the conditioning image from the input camera pose. KID and MMD-CD are ×10³; COV-CD is in %.

MethodObs. FIDObs. KIDUnobs. FIDUnobs. KIDUnobs. COV-CDUnobs. MMD-CDInput-view PSNRInput-view SSIMInput-view LPIPS
GaussianGPT-AR84.6737.8592.6943.0264.3031.5418.60.6800.289
TriFold60.3818.2067.8121.4385.2412.7524.50.8570.201
Δ (TriFold − GaussianGPT-AR)−24.29−19.65−24.88−21.59+20.94−18.79+5.9+0.177−0.088

As shown in Table R5, TriFold achieves 24.5 dB PSNR, 0.857 SSIM, and 0.201 LPIPS at the conditioning viewpoint, indicating better consistency with the input image than the GaussianGPT AR baseline. Separately, the unobserved-region appearance and geometry metrics support improved completion quality under the same evaluation protocol.

We have incorporated these quantitative and qualitative comparisons into Table 2, Fig. 5, Fig. 15, and Appendix A.6 of the revised manuscript.

In the revised manuscript:

Lines 423-473:

Single-image conditioning. Given a single RGB image, TriFold generates a 3D Gaussian scene using a latent flow-matching model. We compare against an image-conditioned GaussianGPT AR model trained on InteriorGS using the same conditioning images and evaluation split. Appendix A.4 details its tokenizer and implementation mechanisms. To evaluate consistency with the input image, we render each generated scene from the input camera. TriFold achieves 24.5,dB24.5\\,\mathrm{dB} PSNR, 0.857 SSIM, and 0.201 LPIPS, compared with 18.6 dB, 0.680, and 0.289 for GaussianGPT (Table 2). To evaluate completion beyond the input view, we back-project its pixels into 3D using depth and reproject them into novel views to identify unobserved regions (Appendix A.6). In these regions, TriFold reduces FID from 92.69 to 67.81 and increases COV-CD from 64.30% to 85.24%. Figures 5 and 15 show examples from novel views.

AuthorsOpenReview ↗

Response to Reviewer MRST (Part 2/2)

W2: The poor reconstruction results of some baselines in Table 1 need further explanation. Appendix A.4 and Figure 9 show that Can3Tok and DiffGS produce similar low-quality scenes from different inputs, raising questions about their training and convergence under the current setup. Can3Tok also requires reducing each input scene to 40,000 Gaussians. The selection procedure and the rendering quality after this reduction should be clarified. Without this analysis, it is difficult to distinguish limitations of the representations from errors introduced by input reduction or difficulties adapting the methods to the current data.

To W2: Baselines training, input reduction, and reconstruction quality. Appendix A.4 specifies that Can3Tok, DiffGS, GaussianGPT, and TriFold are trained from scratch on the same scene-level dataset and split for 80K optimization steps, and evaluated on the same held-out scenes and rendering views. Thus, the reconstruction comparison controls training-domain differences and evaluation conditions.

Can3Tok input selection. The original Appendix A.4 inaccurately stated that each scene was globally downsampled to 40K Gaussians; all reported Can3Tok results use the windowed protocol now described in Appendix A.4 and Fig. 8. Can3Tok's own pipeline fixes 40K Gaussians per sample by growing a single spatially coherent region around a salient seed (Gao et al., 2025), which would cover only a small fraction of our 8 m blocks (median ≈683K Gaussians, Fig. 7). We therefore keep Can3Tok's spatially coherent 40K selection but tile the entire block into such windows, so that no input Gaussian is discarded. Each window is encoded and decoded with the official implementation, and the outputs are de-normalized and stitched back into the full block; we verified the per-window de-normalization and stitching. As shown in Fig. 8, the 40K input windows retain recognizable local structure and appearance before encoding, whereas Can3Tok's reconstructions of the same windows remain coarse. Together with the training curve in Table R6, which plateaus between 60K and 80K steps, this indicates that the degradation arises in Can3Tok's encoding–decoding under our scene-level setting rather than from input selection or insufficient training.

Q2: What training and convergence checks were performed for Can3Tok and DiffGS? For Can3Tok, how were the 40,000 Gaussians selected from each pre-fitted scene? If available, please provide renderings or quality measurements after this reduction and before encoding. These would help separate errors introduced by input reduction from those introduced by the reconstruction model.

To Q2: Training convergence checks and Can3Tok 40K input evaluation.

Thank you for raising this concern. Both Can3Tok and DiffGS were trained from scratch using their official implementations for 80K optimization steps. To monitor convergence, we evaluate their checkpoints on the test set at fixed training intervals, using the same test scenes, camera viewpoints, and ground-truth reference renderings throughout training. At each checkpoint, we reconstruct the test scenes and compute PSNR under an identical evaluation protocol. As shown in Table R6, the PSNR changes between 60K and 80K steps are small for both methods, indicating that their reconstruction performance has largely stabilized by 80K steps under the adopted training configuration. The 40K selection procedure and the renderings of the selected windows before encoding are described in our response to W2 (Fig. 8).

Table R6: PSNR convergence of Can3Tok and DiffGS across different training steps.

stepsCan3Tok PSNRDiffGS PSNR
step=20k2.0310.30
step=40k3.8911.51
step=60k4.4812.28
step=80k4.5312.43
Δ (60K → 80K)+0.05+0.15
Reviewer_sParOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2(+1)
Soundness
3(+1)
Read review
Paper summary

The paper proposes TriFold, a triplane-based latent representation and VAE approach for 3D Gaussian scene generation. The encoder of the TriFold first projects context-aware features from PTv3, aggregating features of each cell by attention with learnable queries. Then, sparse cross-plane attention is proposed to propagate information across the planes. The decoder reconstructs 3D Gaussian by occupancy predictor and learnable queries on each cell. The VAE framework is trained with occupancy supervision, Gaussian decoder supervision, and phantom-voxel supervision. Latent flow-matching model is implemented on the TriFold VAE. On InteriorGS and SceneSplat-49K dataset, TriFold showed the best VAE reconstruction performance compared to baseline methods including Can3Tok, DiffGS, GaussianGPT. Ability of conditional generation is demonstrated qualitatively and quantitatively. Ablation study validates each design choice in the VAE architecture.

Critical strengths
  1. Most of the motivations and design choices of TriFold and each of its component make sense and are demonstrated through experiments. Application of triplane is technically different from previous approaches.

  2. It shows the best Gaussian scene reconstruction performance compared to the baseline methods. The qualitative results look much better than previous results visually.

Critical weaknesses
  1. While the authors argue that unconstrained and variable-length Gaussian is important (line 82-83), the capacity and maximum decoded size of the latent representation might be limited by KK. Although K=24K=24 was a good empirical choice in the experiment set, it might be insufficient for larger scenes.

  2. Training dataset for baseline methods is unclear. According to the paper, TriFold seems to be evaluated on the in-domain dataset. On the other hand, the other methods (e.g. Can3Tok, DiffGS, GaussainGPT) may not be trained on the same dataset. In this case, quantitative comparison is unfair and it is unknown whether superior reconstruction performance comes from the dataset.

  3. Computational cost of VAE methods, especially the size of the latents, should be compared. If the size of TriFold latent were significantly larger than other approaches, the reconstruction performance might have come from the latent size, not from its architectural design.

  4. Comparison of conditional generation is missing. Although showing the best ability reconstruction ability, the generation can be possibly more difficult compared to the previous latent representations as TriFold suggested triplane-based latent space.

  5. It is difficult to follow the details of VAE architecture. While components such as cross-plane attention look important, all of their details are in appendix, making the reader complex to understand the architecture. Also, figure 2 is difficulty to understand each component. It would be better if each components of VAE is illustrated in more details with equations or figures.

Questions for authors
  1. For occupancy supervision, why is a formulation of regression used instead of classification such as binary cross entropy? A choice of τocc\tau_{occ} due to the regression formulation looks arbitrary and unnecessary.

  2. Prior approach such as GaussianGPT uses vector quantization to be utilized for autoregressive generation. Would TriFold still outperform GaussianGPT when vector quantization is applied, demonstrating potential ability of TriFold for autoregressive generation?

6 comments
AuthorsOpenReview ↗

Response to Reviewer sPar (Part 1/4)

We sincerely thank the reviewer for the constructive comments and for recognizing TriFold’s motivation, design choices, and reconstruction quality. In response to your concerns, we have clarified the representation’s capacity limits and baselines training protocol, and strengthened the evidence through 1) computational cost, 2) conditional generation comparisons with a GaussianGPT-based image-conditioned AR model, 3) a comparison between occupancy regression and soft-target BCE, and 4) VQ experiments examining discrete reconstruction and the potential for autoregressive modeling. We have also redesigned Figure 2 and moved the core architectural formulations into the main text. We address each point below.

W1: While the authors argue that unconstrained and variable-length Gaussian is important (Lines 82-83), the capacity and maximum decoded size of the latent representation might be limited by KK . Although KK=24 was a good empirical choice in the experiment set, it might be insufficient for larger scenes.

To W1: Capacity limits and scalability to larger scenes

Thank you for raising this point. We agree that K and the spatial grid resolution of triplane features bound scene-level representation capacity, a fundamental challenge prevalent across existing approaches (e.g., TRELLIS and GaussianGPT are similarly bounded by discretized grids, while Can3Tok uses a fixed input budget per window in our baseline configuration). To state our scope more precisely without implying unbounded capacity, we have removed "unconstrained" in Lines 82–84, emphasized "content-adaptive local Gaussian allocation during decoding" in Lines 110–113, and clarified in Appendix A.6 that scaling to substantially larger or denser scenes may require adjusting the grid resolution or local budget KK .

Within this finite capacity boundary, our core advantage lies in how flexibly the Gaussian budget is allocated across space. Rather than enforcing a rigid, uniform primitive count per voxel or resampling the entire scene to a fixed global budget, our decoder dynamically predicts which voxels to activate and how many candidates to retain locally (Sec. 3.3). Because KK acts solely as a local upper bound rather than a fixed output count, TriFold concentrates primitives on geometrically complex regions. Empirically, 97.8% of occupied voxels contain ≤24\le 24 Gaussians (Fig. 7), and increasing KK to 32 yields marginal gains (PSNR: 25.83→25.9625.83 \rightarrow 25.96 dB; LPIPS: 0.096→0.0920.096 \rightarrow 0.092, Table 14).

In the revised manuscript:

  • Lines 82–84: "Consequently, an open challenge remains: how can we provide generative backbones with a regular, fixed-layout latent space, while preserving the variable-length nature of native Gaussian representations at both the input and output?"
  • Lines 110–113: "Native scene-level 3D Gaussian representation learning. We introduce TriFold, a variational autoencoder that learns structured representations of Gaussian scenes with variable cardinality and nonuniform spatial density by folding irregular primitives into fixed-layout latents while supporting content-adaptive local Gaussian allocation during decoding."

W2: Training dataset for baselines methods is unclear. According to the paper, TriFold seems to be evaluated on the in-domain dataset. On the other hand, the other methods (e.g. Can3Tok, DiffGS, GaussianGPT) may not be trained on the same dataset. In this case, quantitative comparison is unfair and it is unknown whether superior reconstruction performance comes from the dataset.

To W2: Fairness of baselines training and evaluation

Thank you for emphasizing experimental fairness. We confirm that all baselines (Can3Tok, DiffGS, and GaussianGPT) were trained from scratch following their official implementations, using the exact same InteriorGS + Aria training splits (9:1 split prior to cropping to prevent data leakage), and evaluation protocols as TriFold (Sec. 4.1 and Appendix A.4).

Specifically, we aligned the experimental setup across three aspects:

  • Training setup: All models were trained for 80K steps using their official implementations and method-specific optimization settings, with the spatial adaptations documented in Appendix A.4.
  • Scene inputs and windowing: TriFold, DiffGS, and GaussianGPT used identical scene inputs. Since Can3Tok uses a fixed 40K-Gaussian input budget in our baseline configuration, we split each evaluation block into spatially coherent 40K-Gaussian local windows and merged the reconstructed windows for full-block evaluation.
  • Evaluation views: All methods were trained on the same training scenes and evaluated on the same held-out scenes using identical camera viewpoints and ground-truth reference renderings.

We have moved these protocol details from Appendix A.4 into Sec. 4.1 of the main paper to make the fairness of our experimental setup immediately clear.

AuthorsOpenReview ↗

Response to Reviewer sPar (Part 2/4)

W3: Computational cost of VAE methods, especially the size of the latents, should be compared. If the size of TriFold latent were significantly larger than other approaches, the reconstruction performance might have come from the latent size, not from its architectural design.

To W3: Computational cost of VAE.

We thank the reviewer for raising this critical point. To directly isolate the effect of latent capacity from our architectural design, we trained a single-slot TriFold variant (Q=1) from scratch under the exact same multi-dataset protocol as Table 1. In Table R1, we compare the latent footprints, parameter counts, inference latencies, and reconstruction fidelity across the native 3DGS autoencoders:

Table R1. Representation cost and reconstruction quality of 3DGS autoencoders. All methods are trained on the same InteriorGS + Aria training split; reconstruction metrics are on the InteriorGS test set (Table 1). Latency is measured for one 500K-Gaussian scene on an NVIDIA RTX A6000. For Can3Tok, the latent size is 0.031 MiB per 40K-Gaussian window, and the total inference latency is 312 ms for the full 500K-Gaussian block (approximately 13 windows). DiffGS: Adapted to use a 2D triplane latent representation with 16 channels per plane, instead of the original 1D latent vectors. For GaussianGPT, N_active = 81029 represents the average number of active voxels across the evaluated scenes, and 64 denotes the channel dimension. CD is ×10³.

MethodLatent ShapeLatent (MiB)Parameters (Million)Latency (ms)PSNR↑SSIM↑LPIPS ↓CD ↓
Can3Tok512×32 (per window)0.0311081.553124.530.0120.92010.96
DiffGS48×256×2566.035.4776212.430.7190.3743.21
GaussianGPTN_active×649.9176.22102721.400.8560.1572.62
TriFold (1 slot)1×48×256×2566.054.24182125.210.9010.1042.27
TriFold (4 slots)4×48×256×25624.054.34214026.550.9130.0882.23
  • Latent size vs. encoder–decoder architecture. We conduct two controlled comparisons to disentangle the effects of latent capacity and architecture. First, despite using the same 48 × 256 × 256 continuous latent, TriFold (1 slot) achieves 25.21 dB PSNR, substantially outperforming the adapted DiffGS baseline (12.43 dB; Table 17). Second, keeping GaussianGPT's sparse voxel grid and VQ bottleneck unchanged, replacing only its encoder and decoder with ours improves PSNR from 21.40 to 24.36 dB (Table 15, Rows 0–1). Moreover, doubling TriFold's latent channel width (Cz = 16 → 32) yields only a 0.06 dB gain (Table 13), whereas increasing the number of slots (Q = 1 → 4) improves PSNR by 1.43 dB (Table 12). These results suggest that latent size alone does not account for our reconstruction gains.

  • Architectural gains at fixed latent size. Our architectural ablations (Table 7) further confirm the importance of the proposed design. Under an identical latent budget, query-based aggregation improves PSNR from 24.60 to 25.83 dB over mean pooling, while our variable-length set decoder improves PSNR from 23.09 to 25.83 dB over a fixed MLP head. These results demonstrate that the architectural design, rather than merely increasing latent capacity, plays a key role in reconstruction quality.

We have incorporated these results and this capacity analysis into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer sPar (Part 3/4)

W4: Comparison of conditional generation is missing. Although showing the best ability reconstruction ability, the generation can be possibly more difficult compared to the previous latent representations as TriFold suggested triplane-based latent space.

To W4: More generation comparison

Thank you for raising this thoughtful point. We agree that higher reconstruction fidelity does not automatically guarantee easier generative modeling, particularly when generating structured triplane latents.

Quantitative comparison on conditional generation.

We added a direct comparison of single-image-conditioned scene generation on InteriorGS. We selected GaussianGPT because it achieved the highest reconstruction PSNR among the baselines in Table 1 (21.40 dB). Can3Tok and DiffGS produced coarse reconstructions under our scene-level setup (Fig. 10).

We adapted GaussianGPT’s autoregressive model for single-image conditioning and used a stride-2 tokenizer to keep the scene sequences tractable for AR training. Table R2 compares this 4.74B-parameter AR baseline with TriFold’s 1.1B-parameter latent flow model. We evaluated both systems on the same InteriorGS split using the same conditioning images, evaluation views, and depth-reprojection masks. Appendix A.4 further describes the implementation details. We exclude Can3Tok and DiffGS from stage-2 comparisons because their stage-1 autoencoders converge to overly coarse reconstructions at scene scale (Fig. 10), making it difficult to separate generator errors from stage-1 bottlenecks.

Table R2: Quantitative comparison of single-image-conditioned scene generation on InteriorGS. Both methods are evaluated under the identical split, conditioning views, and 3D depth-reprojection masks. KID and MMD-CD are ×10³; COV-CD is in %.

MethodObs. FID ↓Obs. KID ↓Unobs. FID ↓Unobs. KID ↓Unobs. COV-CD ↑Unobs. MMD-CD ↓
GaussianGPT-AR84.6737.8592.6943.0264.3031.54
TriFold (Ours)60.3818.2067.8121.4385.2412.75
Δ (TriFold − GaussianGPT-AR)−24.29−19.65−24.88−21.59+20.94−18.79

As shown above, TriFold outperforms GaussianGPT AR model across all metrics, lowering unobserved FID by 24.88 points (67.81 vs. 92.69) and reducing unobserved geometric error (MMD-CD) by nearly 60% (12.75 vs. 31.54).

We have incorporated these generation comparisons and discussions into Sec. 4.3 of the revised manuscript.

W5: It is difficult to follow the details of VAE architecture. While components such as cross-plane attention look important, all of their details are in appendix, making the reader complex to understand the architecture. Also, figure 2 is difficulty to understand each component. It would be better if each components of VAE is illustrated in more details with equations or figures.

To W5: Clarity of the VAE architecture and framework illustration

Thank you for this helpful feedback. We agree that the original Figure 2 was overly dense and that keeping core mechanisms in the appendix made the architecture harder to follow. We have updated the main paper in two ways to make the VAE description clearer:

  • Redesigned Figure 2 with four components: We restructured Figure 2 into four clear sub-diagrams: (a) the end-to-end VAE training pipelines with occupancy-guided Gaussian decoding, (b) the triplane encoder, explicitly illustrating how learned queries aggregate projected Gaussian features within each plane cell and how sparse cross-plane attention exchanges information across orthogonal planes via shared Gaussian projections, (c) the ResNet-based variational bottleneck mapping features into the fixed-layout latent, and (d) the conditional scene generation pipelines.

  • Expanded Sec. 3.2 with core formulations: We moved the mathematical formulations and step-by-step explanations of cell-wise query aggregation, sparse cross-plane attention, and the variational bottleneck from Appendix A.2 directly into Sec. 3.2 (Lines 210–243, with references to Fig. 2 b–c), leaving only low-level layer hyperparameters in the appendix.

These revisions clearly walk the reader through how TriFold encodes irregular 3D Gaussian primitives into a structured triplane latent. We sincerely thank the reviewer for helping us significantly improve the clarity of our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all three reviewers for their constructive feedback and for recognizing TriFold’s motivation, adaptive representation, and strong reconstruction performance. TriFold imposes structural regularity in the latent space while keeping decoded Gaussians content-adaptive, mapping variable-cardinality Gaussian sets into a fixed-layout triplane and decoding a content-adaptive number of primitives per occupied voxel. To address all concerns raised during review, we have carefully revised our manuscript with key enhancements including 1) a rigorous computational cost and latent-capacity profiling (Table R1); 2) a direct single-image conditional generation and consistency comparison against a GaussianGPT AR baseline (Tables R2 & R5); and 3) cross-architecture representation ablations demonstrating our framework's generality (Table R4). Below is a summary of our specific responses to each reviewer:

● Reviewer sPar — Capacity, computational cost, and architectural clarity. We clarified the role of K and grid resolution, compared latent footprints, model parameters, and inference latency across autoencoders (Table R1), and added an occupancy-objective ablation comparing regression with soft-target BCE (Table R3). To decouple architectural design from latent capacity, we replaced GaussianGPT's encoder and decoder with ours while keeping its sparse voxel layout and VQ bottleneck unchanged, improving PSNR from 21.40 to 24.36 dB (Table R4). We also redesigned Figure 2 and moved the core VAE formulations into Section 3.2.

● Reviewer MRST — Baseline validity and conditioning consistency. We verified baseline training convergence across optimization steps (Table R6) and added side-by-side visualizations of Can3Tok's 40K-Gaussian input windows and reconstructions (Figure 8). To evaluate generation against an existing representation, we adapted GaussianGPT to single-image conditioning under the same protocol, where TriFold improves input-view consistency from 18.6 to 24.5 dB PSNR (Table R5).

● Reviewer YA1A — Methodological contribution and generation evaluation. We clarified our representation-learning contribution via cross-architecture ablations, showing that even on an identical 48 x 256 x 256 continuous triplane latent, TriFold achieves 25.21 dB PSNR vs. 12.43 dB for DiffGS (Table R1, Table R4), and consolidated all model sizes and training details (Appendix A.10). We also clarified in Section 4.3 that point-cloud conditioning evaluates missing-attribute recovery from complete geometry, while in single-image generation TriFold lowers unobserved-region FID from 92.69 to 67.81 compared to image-conditioned GaussianGPT (Table R2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.