acceptodds
Under review as a conference paper at ICLR 2027

Beyond Global Latents: Chunk-Based Sparse Grid VAE for Scalable 3D Modeling

Abstract

Sparse voxel grids preserve the spatial structure needed for detailed 3D reconstruction, but their memory still grows rapidly with resolution as active surface cells increase. We introduce ChunkVAE, a sparse grid variational autoencoder organized around local chunks rather than a global latent volume. Local learned operators permit independently chosen encoder and decoder partitions and allow inference chunk sizes to differ from training. Two complementary data operators make this flexibility practical: Balanced Binary Object Partitioning distributes active cells while limiting replicated overlap, while S-Curve weighted stitching attenuates unreliable boundary features when assembling a global latent or reconstruction. Across three object benchmarks, ChunkVAE is competitive with or better than strong baselines from to ; smaller chunks lower peak allocated memory and shorten per-chunk compute, enabling faster parallel inference. Stable stitched latents and improved image to 3D metrics indicate that local compression can scale geometry while retaining the global interface required downstream.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sparse Grid VAE, Chunk, Scalable 3D Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3oELOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ChunkVAE, a sparse grid VAE organized around local chunks rather than a global latent volume. The core idea is to partition the global sparse voxel grid into overlapping local chunks, encode and decode each chunk independently, then stitch the results into a global latent or reconstruction via deterministic operators. Two such operators are proposed: BBOP (Balanced Binary Object Partitioning) achieves load-balanced chunking through recursive median splitting along the longest axis, and S-Curve weighted stitching assigns reliability weights based on distance from chunk core to suppress boundary artifacts. Experiments on Toys4K, ABO, and HQM benchmarks from 512³ to 1536³ demonstrate state-of-the-art or competitive reconstruction quality. The paper also shows that training and inference chunk sizes can be decoupled, encoder and decoder partitions can be chosen independently, and the stitched latent preserves global consistency.Notably, the paper adopts a largely engineering-oriented strategy, combining practical system design choices such as chunk-based parallelism, deterministic partitioning, and weighted stitching rather than novel architectural innovations, and achieves strong empirical results through this pragmatic approach.

Critical strengths

s1.Clear motivation and practical value. Memory scaling with resolution is a real bottleneck for sparse grid VAEs. ChunkVAE reduces peak allocated memory from 16.22 GB to 4.57 GB (71.8% reduction) under a <1M cell budget while maintaining competitive reconstruction quality. This trade-off is meaningful for practical deployment.

s2. Elegant training-inference decoupling. Training and inference can use different chunk budgets, and encoder and decoder can adopt different partitions, all without retraining. Table 5 validates this flexibility thoroughly. This is practically useful for memory-constrained inference devices.

Critical weaknesses

W1.Limited conceptual novelty. Chunk-based processing has precedents in 2D image VAEs (e.g., the tiling mechanism in FLUX.1) and voxel rendering. Migrating it to 3D sparse grids with BBOP and S-Curve stitching is reasonable, but the overall "divide and stitch" paradigm is not surprising. BBOP is essentially recursive median splitting along the longest axis, and S-Curve is a deterministic distance weight. Both are sound but relatively straightforward engineering choices. For an ICLR contribution, the conceptual insight is somewhat lacking.

W2. Generation experiments are merely pipeline substitution. The image-to-3D experiments (Table 3) directly reuse TRELLIS.2's flow-matching pipeline with only the VAE swapped. The improvements are marginal (ULIP-I: 0.134 vs 0.130, Uni3D-I: 0.318 vs 0.317), making it hard to attribute gains to ChunkVAE itself rather than pipeline adaptation. More critically, the paper does not report training details of the generative model in ChunkVAE's latent space (was it retrained? were hyperparameters tuned?), does not show the impact of chunk boundaries on diffusion or flow sampling, and does not provide perceptual quality comparison against generation directly in TRELLIS.2's o-voxel latent.

W3.No support for non-manifold or open surfaces. The paper relies on flood filling to convert meshes to watertight surfaces, meaning it cannot handle non-manifold geometry or open surfaces (e.g., cloth, paper). This is an inherent limitation of SDF representation, but the paper's title claims "scalable 3D modeling" without discussing this applicability boundary

W4.Only 8× spatial compression is demonstrated. The paper fixes 8× spatial compression (3 downsampling stages) and does not explore chunk behavior under other compression ratios (4×, 16×). Active cell density and distribution differ significantly across compression ratios, and more evidence is needed for the robustness of BBOP and S-Curve.

W5.No analysis of partition strategy impact on VAE training. Table 4 compares USP, RK-2, and BBOP partition quality metrics (CSCV, CRF, Success Ratio), but uses the same trained VAE, only swapping the partition at inference time. The paper answers "which partition strategy is better" but does not answer "whether VAEs trained with different partition strategies differ."

Questions for authors

Q1.Why not consider part-based decomposition of objects instead of (or in addition to) chunking to reduce the computational burden of the VAE? Given that numerous part-level datasets and pretrained models are already available, a part-aware representation could better preserve local consistency.

Q2. At 1536³ resolution, how expensive is the SDF extraction step (mesh-to-watertight SDF voxelization) in terms of runtime and memory? How much geometric detail is lost during this conversion, especially for thin structures, high-curvature regions, and fine surface features? Is there a quantitative analysis of detail loss as a function of resolution?

Q3. What are the key quality issues in the current VAE reconstruction output, such as holes, non-manifold edges, or boundary artifacts? Does the paper systematically evaluate the prevalence of these defects across different resolutions and geometry types? If such issues exist, what strategies are employed or planned to mitigate them, and how do they compare to post-processing approaches like mesh repair or watertight optimization?

1 comment

W1. Methodological contribution. Our methodological contribution goes beyond the divide-and-stitch paradigm: we make sparse 3D VAEs scalable under independently chosen encoder/decoder partitions and varying training/inference budgets while maintaining consistent global latents. Unlike dense 2D tiling, sparse 3D chunking must account for highly uneven active-voxel distributions, padding-induced replication, and boundary-dependent predictions and pruning. BBOP balances padded active-voxel workloads, while S-Curve stitching reduces unreliable boundary contributions.

W2. Generation architecture and training. Our generator was trained in our latent space for 500K steps. It is not a TRELLIS.2 decoder swap. The TRELLIS reference concerns its two-stage flow-matching framework. We use a FLUX-style sparse DiT with double-stream/single-stream blocks and 2× packing/unpacking. Training uses warm-start, Adam, learning rate 1e-4, total batch 1024 and BF16, with our own hyperparameters. We will clarify this.

The 8× VAE plus 2× packing gives an effective 16× Transformer scale. Sampling uses assembled global latents, without separate chunk trajectories.

The gains are meaningful: +0.004 ULIP-I is four times the 0.001 gap between CraftsMan and Hunyuan3D-2.1/LATTICE in Table 3. +0.001 Uni3D-I is comparable to leading baselines' 0.001–0.002 gaps. Figures 5 and 9 show clearer steps, bookshelves, robot joints and rivets. Table 1 and fixed-VAE ablations establish reconstruction gains independently of generator adaptation; Table 3 demonstrates generation quality.

W3. Surface applicability. Our SDF targets watertight solids, as disclosed in the Limitations section. Watertight geometry is widely used in SDF-based generation, including Hunyuan3D-2.1. Chunking is not tied to this constraint; field-free representations such as TRELLIS.2's O-Voxel could support arbitrary topology.

W4. Compression robustness. BBOP operates on active coordinates and padded voxel budgets, while S-Curve stitching uses spatial distances. Neither design is tied to a specific compression ratio or number of downsampling stages. Different compression ratios may change latent density and receptive fields, requiring corresponding adjustments to chunk budgets and padding sizes, rather than changes to the partitioning and stitching algorithms.

W5. Training partitions. USP and RK-2 already expose problems at inference (Introduction; Section 3.2). USP produces uneven, under-populated chunks vulnerable to decoder pruning and mesh fractures. RK-2 creates overlapping boxes and repeated computation. Training inherits these problems, motivating BBOP. Table 4 quantifies these issues (CSCV, CRF, Success Ratio): on HQM1536, BBOP reduces CSCV from 0.876 to 0.150 versus USP and CRF from 2.584 to 1.702 versus RK-2, with 100% success versus USP's 80%.

Q1. Part-based decomposition. Parts can still exceed memory limits. BBOP controls padded workloads without part annotations; large parts can also be chunked and stitched.

Q2. Preprocessing cost and detail preservation. Preprocessing runs offline once per asset and resolution. With LBVH-CUDA, our watertight conversion takes around 10–20 s at 1536³, averaging approximately 40 GB GPU memory. Memory depends on surface area and active-voxel counts.

We evaluate against original ABO/Objaverse meshes. CD measures geometric deviation (×10³), ANC normal fidelity (%) and F1 surface coverage (%, threshold 0.01).

MethodABO CD ↓ABO ANC ↑ABO F1 ↑Objaverse CD ↓Objaverse ANC ↑Objaverse F1 ↑
Dora-UDF [1]5.9699.4492.507.6898.5793.39
Visibility-check [2]13.6399.3582.208.9696.4886.10
Our watertight method (1024³)7.0799.6090.612.2299.0595.30
Our watertight method (2048³)3.5499.6594.411.3099.5197.38

Doubling resolution reduces our CD by 50% on ABO and 41% on Objaverse. At the reported settings, our 2048³ result leads both baselines on all metrics and datasets. Detail restoration recovers thin structures and sharp features while preserving watertight connectivity.

Q3. Reconstruction defects. The main risks are holes in thin structures and boundary seams. Our two-layer SDF band on both sides improves continuity; overlap and S-Curve suppress unreliable boundary predictions. Figure 6 isolates stitching and shows fewer boundary artifacts. Figures 4 and 8 show continuous strings, spokes and mechanical details across 512³–1536³. Mesh repair can close holes but alter detail; TRELLIS.2's export workflow applies remeshing and simplification, which can smooth fine features and compromise thin structures. Decoding and stitching improve continuity, preserving detail with less reliance on repair.

[1] Chen et al. Dora. CVPR 2025.

[2] Zhang et al. CLAY. ACM TOG 2024.

Reviewer_xp65OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ChunkVAE, a Sparse Voxel VAE for high-resolution 3D generation. By adopting local network operators, the model enables chunk-wise processing for both the encoder and decoder independently, and allows different chunk partitions to be used during training and inference. In addition, the authors propose BBOP for load balancing, together with S-Curve Weighted Stitching to mitigate boundary artifacts across chunks.

Experiments demonstrate that this method achieves SOTA reconstruction performance at resolutions ranging from 512 to 1536, and can reduce peak memory consumption via chunking.

Critical strengths

First, this paper addresses the memory bottleneck of high-resolution Sparse Voxel VAE, and the research problem is of practical significance. The design of independent chunking for the encoder and decoder, as well as the decoupling of chunk sizes between training and inference, has certain practical value.

Second, the paper achieves favorable reconstruction results across multiple datasets and resolutions, and performs particularly well on HQM, a dataset with complex geometries. BBOP also demonstrates good load balancing and reconstruction stability.

Critical weaknesses
  1. This work is mainly motivated by addressing the memory and computation bottlenecks of high-resolution Sparse Grid VAE. However, Table 2 only reports the inference time and memory overhead at 5123512^3, lacking efficiency comparisons at higher resolutions such as 102431024^3 and 153631536^3. At 5123512^3, chunking reduces memory overhead at the cost of increased single-GPU inference time. The reported per-chunk time in the paper is only the idealized parallel computation time, rather than the real end-to-end parallel inference latency. Therefore, the current experiments are insufficient to fully demonstrate the practical scalability advantage of ChunkVAE in high-resolution scenarios. The authors are suggested to supplement the memory–latency–quality trade-off and real parallel inference performance under high resolutions.

  2. This work adopts local operators as the core design for implementing chunk-wise VAE. Nevertheless, the existing TRELLIS.2 Geometry VAE already primarily employs local sparse convolutions without global self-attention. Therefore, such existing architectures in principle have the foundation to realize chunk-wise computation via proper padding and stitching. The paper fails to sufficiently elaborate the essential differences between its VAE architecture and existing local VAEs. It also lacks ablation baselines that directly apply chunk-wise processing to existing methods such as TRELLIS.2. In addition, BBOP bears considerable resemblance to the classic median-split KD-tree partitioning, and S-Curve Stitching is similar to common feathering/window-based blending approaches. The paper lacks direct comparisons against these classic methods and does not adequately quantify the individual contributions of the two modules to the final reconstruction quality. Hence, it remains unclear how substantial the technical advantages of this work are over existing local VAE architectures and classic chunking and blending strategies.

  3. This paper’s core assumption is that consistent global latents can be constructed from independently processed chunks using local operators and stitching. However, local operators do not automatically guarantee consistency between chunk-wise encoding and global encoding. This property also depends on the effective receptive field, padding size, and specific computation operations of the network. The paper does not explicitly analyze whether the proposed network satisfies these requirements. Furthermore, S-Curve Stitching mainly smooths boundaries via spatial weighting and cannot directly guarantee the consistency of latent features or pruning decisions across different chunks. Although Tables 5–6 present reconstruction and classification results under different chunk budgets, these experiments only provide relatively indirect validation of consistency. The authors are suggested to directly compare latent discrepancy, boundary reconstruction error, and the consistency of final reconstruction results under different chunk partitions, so as to more fully verify the core assumption of their method.

  4. 3D VAEs are required not only to achieve high-quality geometric reconstruction but also to provide compact, easily modelable latent representations for downstream DiTs. However, this paper mainly focuses on the VAE reconstruction quality and the efficiency of chunk-wise processing, lacking a systematic analysis of the computational overhead for downstream generative models. Although ChunkVAE reduces the peak memory consumption in the VAE stage, it still needs to assemble the complete global latent for global modeling by the DiT. As the resolution increases, the growth in the number of latent tokens may introduce higher training and inference overhead for the DiT, thereby diminishing the efficiency gains obtained from VAE chunking. Moreover, superior VAE reconstruction quality does not necessarily imply that the latents are easier for DiTs to learn or generate. While Table 3 demonstrates performance improvements for image-to-3D tasks, the gains are relatively modest. The paper also omits analysis of the relationship among latent token count, DiT training cost and generation quality. Therefore, the authors are suggested to further evaluate the practical impact of ChunkVAE on downstream DiTs. For instance, compare generation quality, training efficiency and the computational overhead of the full pipeline under an identical DiT compute budget.

Questions for authors

as above information.

1 comment

1. High-resolution efficiency and parallel latency. On H20 GPUs, we measure time, memory and F-score on the same HQM objects. Encoder budgets of 1M/2M/4M voxels use decoder budgets of 2K/4K/8K latent voxels.

ResolutionBudgetPeak (GiB) ↓Execution (s) ↓F-score ↑
1024³1M3.3724.4699.969
1024³2M5.7716.6899.971
1024³4M10.5512.6299.969
1536³1M5.4547.2499.990
1536³2M6.0940.2099.990
1536³4M10.5731.7499.990

F-score is scaled by 100. Reducing the budget from 4M to 1M saves 68.1%/48.4% peak memory at 1024³/1536³, with essentially unchanged quality.

Measured parallel execution includes scheduling and transfers:

ResolutionBudget1 GPU (s)2 GPUs (s)4 GPUs (s)8 GPUs (s)
1024³1M24.4616.0610.548.48
1536³1M47.2426.4015.8911.19

Four GPUs provide 2.32×/2.97× speedups at 1024³/1536³; eight GPUs provide 2.88×/4.22×. F-score changes by less than 0.003 points. Parallel gains increase with resolution while preserving quality.

2. Local VAEs and classic strategies. TRELLIS.2's local convolutions provide a basis for chunk-wise computation. However, locality alone does not address uneven sparse workloads or inconsistent boundary predictions. BBOP and S-Curve address these issues, enabling independent encoder/decoder partitions and varying training/inference budgets while preserving reconstruction quality.

BBOP adapts KD-tree splitting to sparse VAEs by budgeting active voxels including padding. Table 4 evaluates BBOP against other partition strategies on HQM1536: 100% reconstruction success versus 80% for uniform partitioning, and a replication factor of 1.702 versus 2.584 for recursive K-means.

With the VAE and partition fixed, we measure boundary CD around internal partition faces:

StitchingBoundary CD at 1024³ ↓Boundary CD at 1536³ ↓
Full padded averaging1.9721.281
Unique selection1.0760.689
Restricted averaging0.8870.553
Linear feathering (p=1)0.8960.561
Cosine window0.8930.558
S-Curve (p=2)0.8770.547

S-Curve reduces boundary error by 55.5–57.3% versus full padded averaging and 18.5–20.6% versus unique selection. Linear and cosine windows perform similarly. Downweighting unreliable padded predictions is key; p=2 gives the lowest error at both resolutions.

3. Consistency under repartitioning. Three steps address consistency:

  1. Spatial reference. Sparse coordinates and attention-window assignments retain global indices across chunks (Section 3; Appendix A).
  2. Boundary context. Padding supplies neighboring context; S-Curve downweights predictions near padded exteriors where context is truncated.
  3. Direct verification. On seven 512³ HQM objects, we vary budgets. Latents and support use a separate unchunked reference; 4M remains chunked.

Relative latent L2 is 100∥Zc−Zg∥F/∥Zg∥F100\|Z_c-Z_g\|_F/\|Z_g\|_F. ZcZ_c/ZgZ_g are chunked/unchunked encoder features aligned at common sparse voxel coordinates; ∥⋅∥F\|\cdot\|_F is the L2 norm over all aligned voxel features. Results are averaged over objects.

Chunk budgetRelative latent L2 (%) ↓Final support IoU (%) ↑F-score ↑
1M2.41499.70499.574
2M1.18999.85999.625
4M0.14699.92199.646

Latent L2 is 0.146–2.414%, support IoU exceeds 99.70%, and F-score varies by only 0.072 points. Larger chunks reduce latent discrepancy; reconstruction remains stable. This agrees with Tables 5–6: switching encoder/decoder to global processing changes F-score by only 0.07/0.22 points; a classifier trained on global latents loses at most 0.97 percentage points on stitched latents.

4. Downstream DiT cost and quality. To control token growth with resolution while preserving detail, we retain an 8× VAE and add 2× generator packing/unpacking. Packing groups neighboring voxels into coarser tokens; unpacking restores the finer grid. This gives an effective 16× Transformer scale, matching TRELLIS.2.

Token counts on 95 paired HQM objects are:

ResolutionTRELLIS.2 tokensOur packed tokens
512³2,5723,143
1024³11,18113,679
1536³26,11732,380

The remaining 22–24% token gap primarily reflects our thicker SDF band: two voxel layers inside and outside the surface improve geometric continuity. With the same public Sparse DiT backbone (TRELLIS.2) and packing/unpacking for our inputs, training-step overhead is 18–36% at 512³. At 1536³, forward time is 2.609 s versus 1.845 s.

Additional tokens retain more complete geometry; Table 3 demonstrates generation quality. Chunk-wise VAE processing controls memory; generator packing controls sequence length. Together, these designs preserve fine geometry at manageable memory and sequence costs.

Reviewer_KvGbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes training the VAE using local chunks, allowing inference with chunk sizes different from those used during training. Improved image-to-3D results demonstrate the effectiveness of this approach.

Critical strengths
  • Clear motivation. The method is straightforward and easy to understand, with a clear design.
  • Effective experimental results. The method appears to achieve competitive reconstruction and generation results.
Critical weaknesses
  • Limited novelty. The main weakness is its limited novelty. This is more of an engineering design than a novel methodological contribution.
  • Insufficient details. Limited details are provided about the generation stage, making it difficult to assess the effectiveness of the VAE design.
Questions for authors

The VAE appears to use 8× spatial compression, compared with 16× in TRELLIS, making a direct comparison difficult.

From my understanding, this could make training the generation stage more challenging. Does it incur significantly higher training costs than TRELLIS.2?

1 comment

Thank you for your clear and direct comments.

1. Methodological novelty. We agree that chunking itself is not new, our methodological contribution lies in making chunk-based sparse VAEs reliable and scalable under varying spatial partitions and computational budgets. Reliable chunking is a key scaling route for sparse voxel models, since active voxel counts still grow with resolution. The challenge is to preserve latent and reconstruction consistency when partitions change. ChunkVAE combines local learned operators, BBOP to balance active voxels and limit overlap, and S-Curve weighted stitching to suppress unreliable boundary features. This enables independent encoder/decoder partitions and different training/inference chunk budgets.

Our ablations show concrete gains. BBOP raises HQM1536 reconstruction success from 80% with uniform partitioning to 100%. Changing the training chunk budget by 10× changes CD by at most 0.08 at a fixed inference budget. The same VAE trained on 512³ object chunks reconstructs a 3072³ scene without retraining.

2. Generation details. Our generator is a FLUX-style sparse DiT with double-stream and single-stream blocks. It uses 2× packing before the DiT and unpacking afterward. We train it in our latent space for 500K steps with warm-start initialization, using Adam, learning rate 1e-4, total batch 1024, and BF16. Sampling uses 25 steps and CFG 7.5, followed by chunk-wise VAE decoding. We will add these architecture and training details to the paper.

3. 8× versus 16× compression. Our 8× VAE plus 2× packing gives an effective 16× Transformer scale, matching TRELLIS.2. This is a deliberate allocation of compression between the VAE and the generator. The VAE retains a finer latent grid for geometric detail. Packing inside the generator then reduces the token count processed by the DiT. This design preserves a finer reconstruction representation while controlling generation cost.

4. Generation training cost. At the same 16× Transformer scale, token counts still depend on which voxels are active. As described in the paper, our SDF band extends two voxel layers inside and outside the surface to maintain watertight continuity and reduce hole artifacts. It therefore retains more active voxels than TRELLIS.2's surface-only representation. Mean Transformer token counts on 95 paired HQM objects are:

ResolutionTRELLIS.2Ours, after packing
512³2,5723,143
1024³11,18113,679
1536³26,11732,380

After packing, our sequences contain 22–24% more tokens. We isolate their cost using the same public Sparse DiT backbone (TRELLIS.2), including packing/unpacking for our inputs. Small/medium/large denote the 10th/50th/90th percentiles of SDF token count at 512³.

ObjectTRELLIS.2 forward (s)Ours forward (s)TRELLIS.2 step (s)Ours step (s)
Small0.0590.0620.4330.509
Medium0.0980.1270.7210.973
Large0.1970.2561.4702.003

The training-step overhead on this shared backbone is 18–36%, consistent with our longer token sequences. The extra tokens primarily reflect the thicker SDF band retained to improve surface continuity and reduce holes.

In summary, higher compression reduces token counts, but does not remove resource limits at higher resolutions. TRELLIS.2's released 1536³ cascade reuses its 1024³ generator and can lower the requested resolution when token counts exceed its budget. ChunkVAE offers a more flexible route to VAE scaling. Adjustable chunk sizes control encoding and decoding memory without retraining. This allows higher-resolution reconstruction within the available memory budget while retaining a finer latent grid for geometric detail.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.