- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ChunkVAE, a sparse grid VAE organized around local chunks rather than a global latent volume. The core idea is to partition the global sparse voxel grid into overlapping local chunks, encode and decode each chunk independently, then stitch the results into a global latent or reconstruction via deterministic operators. Two such operators are proposed: BBOP (Balanced Binary Object Partitioning) achieves load-balanced chunking through recursive median splitting along the longest axis, and S-Curve weighted stitching assigns reliability weights based on distance from chunk core to suppress boundary artifacts. Experiments on Toys4K, ABO, and HQM benchmarks from 512³ to 1536³ demonstrate state-of-the-art or competitive reconstruction quality. The paper also shows that training and inference chunk sizes can be decoupled, encoder and decoder partitions can be chosen independently, and the stitched latent preserves global consistency.Notably, the paper adopts a largely engineering-oriented strategy, combining practical system design choices such as chunk-based parallelism, deterministic partitioning, and weighted stitching rather than novel architectural innovations, and achieves strong empirical results through this pragmatic approach.
Critical strengths
s1.Clear motivation and practical value. Memory scaling with resolution is a real bottleneck for sparse grid VAEs. ChunkVAE reduces peak allocated memory from 16.22 GB to 4.57 GB (71.8% reduction) under a <1M cell budget while maintaining competitive reconstruction quality. This trade-off is meaningful for practical deployment.
s2. Elegant training-inference decoupling. Training and inference can use different chunk budgets, and encoder and decoder can adopt different partitions, all without retraining. Table 5 validates this flexibility thoroughly. This is practically useful for memory-constrained inference devices.
Critical weaknesses
W1.Limited conceptual novelty. Chunk-based processing has precedents in 2D image VAEs (e.g., the tiling mechanism in FLUX.1) and voxel rendering. Migrating it to 3D sparse grids with BBOP and S-Curve stitching is reasonable, but the overall "divide and stitch" paradigm is not surprising. BBOP is essentially recursive median splitting along the longest axis, and S-Curve is a deterministic distance weight. Both are sound but relatively straightforward engineering choices. For an ICLR contribution, the conceptual insight is somewhat lacking.
W2. Generation experiments are merely pipeline substitution. The image-to-3D experiments (Table 3) directly reuse TRELLIS.2's flow-matching pipeline with only the VAE swapped. The improvements are marginal (ULIP-I: 0.134 vs 0.130, Uni3D-I: 0.318 vs 0.317), making it hard to attribute gains to ChunkVAE itself rather than pipeline adaptation. More critically, the paper does not report training details of the generative model in ChunkVAE's latent space (was it retrained? were hyperparameters tuned?), does not show the impact of chunk boundaries on diffusion or flow sampling, and does not provide perceptual quality comparison against generation directly in TRELLIS.2's o-voxel latent.
W3.No support for non-manifold or open surfaces. The paper relies on flood filling to convert meshes to watertight surfaces, meaning it cannot handle non-manifold geometry or open surfaces (e.g., cloth, paper). This is an inherent limitation of SDF representation, but the paper's title claims "scalable 3D modeling" without discussing this applicability boundary
W4.Only 8× spatial compression is demonstrated. The paper fixes 8× spatial compression (3 downsampling stages) and does not explore chunk behavior under other compression ratios (4×, 16×). Active cell density and distribution differ significantly across compression ratios, and more evidence is needed for the robustness of BBOP and S-Curve.
W5.No analysis of partition strategy impact on VAE training. Table 4 compares USP, RK-2, and BBOP partition quality metrics (CSCV, CRF, Success Ratio), but uses the same trained VAE, only swapping the partition at inference time. The paper answers "which partition strategy is better" but does not answer "whether VAEs trained with different partition strategies differ."
Questions for authors
Q1.Why not consider part-based decomposition of objects instead of (or in addition to) chunking to reduce the computational burden of the VAE? Given that numerous part-level datasets and pretrained models are already available, a part-aware representation could better preserve local consistency.
Q2. At 1536³ resolution, how expensive is the SDF extraction step (mesh-to-watertight SDF voxelization) in terms of runtime and memory? How much geometric detail is lost during this conversion, especially for thin structures, high-curvature regions, and fine surface features? Is there a quantitative analysis of detail loss as a function of resolution?
Q3. What are the key quality issues in the current VAE reconstruction output, such as holes, non-manifold edges, or boundary artifacts? Does the paper systematically evaluate the prevalence of these defects across different resolutions and geometry types? If such issues exist, what strategies are employed or planned to mitigate them, and how do they compare to post-processing approaches like mesh repair or watertight optimization?
W1. Methodological contribution. Our methodological contribution goes beyond the divide-and-stitch paradigm: we make sparse 3D VAEs scalable under independently chosen encoder/decoder partitions and varying training/inference budgets while maintaining consistent global latents. Unlike dense 2D tiling, sparse 3D chunking must account for highly uneven active-voxel distributions, padding-induced replication, and boundary-dependent predictions and pruning. BBOP balances padded active-voxel workloads, while S-Curve stitching reduces unreliable boundary contributions.
W2. Generation architecture and training. Our generator was trained in our latent space for 500K steps. It is not a TRELLIS.2 decoder swap. The TRELLIS reference concerns its two-stage flow-matching framework. We use a FLUX-style sparse DiT with double-stream/single-stream blocks and 2× packing/unpacking. Training uses warm-start, Adam, learning rate 1e-4, total batch 1024 and BF16, with our own hyperparameters. We will clarify this.
The 8× VAE plus 2× packing gives an effective 16× Transformer scale. Sampling uses assembled global latents, without separate chunk trajectories.
The gains are meaningful: +0.004 ULIP-I is four times the 0.001 gap between CraftsMan and Hunyuan3D-2.1/LATTICE in Table 3. +0.001 Uni3D-I is comparable to leading baselines' 0.001–0.002 gaps. Figures 5 and 9 show clearer steps, bookshelves, robot joints and rivets. Table 1 and fixed-VAE ablations establish reconstruction gains independently of generator adaptation; Table 3 demonstrates generation quality.
W3. Surface applicability. Our SDF targets watertight solids, as disclosed in the Limitations section. Watertight geometry is widely used in SDF-based generation, including Hunyuan3D-2.1. Chunking is not tied to this constraint; field-free representations such as TRELLIS.2's O-Voxel could support arbitrary topology.
W4. Compression robustness. BBOP operates on active coordinates and padded voxel budgets, while S-Curve stitching uses spatial distances. Neither design is tied to a specific compression ratio or number of downsampling stages. Different compression ratios may change latent density and receptive fields, requiring corresponding adjustments to chunk budgets and padding sizes, rather than changes to the partitioning and stitching algorithms.
W5. Training partitions. USP and RK-2 already expose problems at inference (Introduction; Section 3.2). USP produces uneven, under-populated chunks vulnerable to decoder pruning and mesh fractures. RK-2 creates overlapping boxes and repeated computation. Training inherits these problems, motivating BBOP. Table 4 quantifies these issues (CSCV, CRF, Success Ratio): on HQM1536, BBOP reduces CSCV from 0.876 to 0.150 versus USP and CRF from 2.584 to 1.702 versus RK-2, with 100% success versus USP's 80%.
Q1. Part-based decomposition. Parts can still exceed memory limits. BBOP controls padded workloads without part annotations; large parts can also be chunked and stitched.
Q2. Preprocessing cost and detail preservation. Preprocessing runs offline once per asset and resolution. With LBVH-CUDA, our watertight conversion takes around 10–20 s at 1536³, averaging approximately 40 GB GPU memory. Memory depends on surface area and active-voxel counts.
We evaluate against original ABO/Objaverse meshes. CD measures geometric deviation (×10³), ANC normal fidelity (%) and F1 surface coverage (%, threshold 0.01).
Doubling resolution reduces our CD by 50% on ABO and 41% on Objaverse. At the reported settings, our 2048³ result leads both baselines on all metrics and datasets. Detail restoration recovers thin structures and sharp features while preserving watertight connectivity.
Q3. Reconstruction defects. The main risks are holes in thin structures and boundary seams. Our two-layer SDF band on both sides improves continuity; overlap and S-Curve suppress unreliable boundary predictions. Figure 6 isolates stitching and shows fewer boundary artifacts. Figures 4 and 8 show continuous strings, spokes and mechanical details across 512³–1536³. Mesh repair can close holes but alter detail; TRELLIS.2's export workflow applies remeshing and simplification, which can smooth fine features and compromise thin structures. Decoding and stitching improve continuity, preserving detail with less reliance on repair.
[1] Chen et al. Dora. CVPR 2025.
[2] Zhang et al. CLAY. ACM TOG 2024.