acceptodds
Under review as a conference paper at ICLR 2027

3Dify-Anything: A Unified Model for Multimodal 3D Generation via Context Pre-Training

Abstract

We introduce 3Dify-Anything, a single model that turns diverse inputs into high-quality 3D assets with fast inference. Inputs can be text, images, sketches, depth maps, point clouds, voxels, bounding boxes, or any combination of these. The model rests on two ideas. The first is an encoder-decoder design that separates understanding the context from generating the shape. A large transformer context encoder reads the input once, and a lightweight rectified-flow decoder generates the 3D latent in a few ODE steps while attending to the encoder's representation. Because only the decoder runs at every step, capacity is concentrated where it is cheap, giving fast inference without sacrificing fidelity. The second idea is context pretraining with cross-modal masking, which makes the encoder strong enough for a small decoder to rely on. The encoder is pretrained with masked token prediction on single- and multi-modal contexts, including the target 3D shape tokens. For multi-modal contexts, one modality is kept visible while the other is heavily masked, so the encoder learns to predict one modality from another. Across a wide range of settings, 3Dify-Anything is competitive with specialized single-task baselines while offering far greater flexibility in how inputs are composed. A compact variant enables near real-time multimodal 3D generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D generation, multimodal generation, masked pretraining, representation learning, encoder-decoder, cross-modal learning, conditional generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_26rvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

3Dify-Anything is a single conditional 3D generator for visual, geometric and textual inputs. A context encoder, pretrained with masking, processes the conditioning once, and a rectified-flow decoder then generates a wavelet-based 3D latent. The proposed pretraining keeps one modality of a pair fully visible and masks the other, so that the encoder learns to predict across modalities. The paper evaluates several single-input tasks and some paired inputs, and ablates the pretraining, the masking objective and the decoder size.

Critical strengths
  1. The paper gives good evidence that the limiting factor is the context representation rather than decoder capacity, which is the core argument for its design. In Table 6, the pretrained model clearly beats a parameter-matched decoder-only model on image, depth and point-cloud conditioning (voxel and text are roughly on par), and training the same encoder from scratch together with the decoder helps little. Table 7 shows that cross-modal masking clearly improves paired-input probes (e.g. image+text AMI 0.355 → 0.592), and Table 16 shows little change from Ours-M to Ours-L, although going from S to M still helps. This is the main reason I lean towards acceptance: it justifies putting capacity into a pretrained encoder instead of a larger decoder.

  2. The design is practical and the results are strong. The large encoder runs once and only a light decoder iterates, which gives about 0.56 s (Ours-S) and 1.47 s (Ours-L) for image-to-3D on one H100 (Tables 1 and 17). On geometric inputs, all three model sizes beat the listed baselines on point-cloud-to-3D IoU on both MAS and GSO (Table 3), and Ours-M/L clearly improve voxel-to-3D IoU over WaLa at the same resolution (Table 4). Fast generation from many input types with one model is useful in practice.

Critical weaknesses
  1. The "any combination" claim is broader than what is shown. The decoder is trained on 22 tasks, all single inputs or pairs (Table 14). Table 5 gives numbers only for voxel, voxel+text and voxel+image, the other trained pairs appear only in figures, and as far as I can tell every combination shown is one of the pairs in Table 14. So the paper supports a unified model for a set of trained input formats, but not generalization to unseen combinations or to more than two inputs. This affects how the contribution is stated more than the method itself, but since it is the headline claim, it should either be tested or narrowed.

  2. The masking objective is the main technical novelty, but its effect on generation from paired inputs, which is its motivation, is not tested. All generation tasks in Table 6 use a single conditioning modality. There, cross-modal masking beats uniform masking on four of the five tasks by modest margins (e.g. image ULIP2 0.340 → 0.364), point cloud slightly favors uniform (IoU 0.914 vs 0.910), and with 250 objects and no variance these gaps are hard to read. Paired inputs are evaluated only through clustering (Table 7), and paired generation (Table 5) only for the final model. It is also not stated which pretraining checkpoint the fixed and uniform encoders in Table 6 use: the probes compare encoders at 300k iterations, while the "ours" row is labeled Ours-S, whose encoder is pretrained for 1M iterations (Sec. 4). This is what keeps my score at marginal acceptance. A paired-input generation comparison of the masking variants would address most of my concern.

Questions for authors
  1. Is the intended claim support for the trained single/paired formats, or generalization to combinations not seen in decoder training? If the latter, can the authors evaluate one held-out combination?

  2. Can the fixed/uniform/cross-modal encoders be compared on a paired generation task (voxel+image or voxel+text) with the same decoder budget? Which pretraining checkpoint does each Table 6 row use?

2 comments

We thank the reviewer for the thoughtful feedback. We are glad the reviewer found the paper's core argument well supported, namely that the limiting factor is the context representation rather than decoder capacity, and found the design practical, with fast generation and strong results on geometric inputs.

The reviewer's concerns are the scope of the "any combination" claim and the effect of cross-modal masking on paired-input generation. We answer the first here and the second in part 2/2; the corresponding changes are in the revised paper (in red).

1. Scope of the "any combination" claim (W1, Q1)

We thank the reviewer for this question, which helped us state the contribution more precisely. Our intended claim is that one model supports the trained input formats, not that it generalises to combinations unseen in training. The reviewer is right that the wording said more. The decoder is trained on 22 formats (9 single inputs and 13 pairs, Table 14), and every combination shown in the paper is one of them. We have narrowed the wording everywhere it appeared: the abstract ("or any combination of these" becomes "or one of 13 supported pairs of these"), the Fig. 1 caption, Secs. 1, 2, 4.1 and 5, and App. B. App. G has a new limitation stating that generalisation to unseen combinations, and to more than two inputs, is not established.

Beyond the trained formats. This is not part of our claim, but the reviewer may find three observations interesting (App. I).

  • Held-out combinations. With Ours-L and the protocol of Table 5, voxel + sketch, a pair the decoder never saw, gives IoU 0.8464 and CD 1.70, and voxel + image + text, which neither the encoder nor the decoder saw, gives IoU 0.8494 and CD 1.65 (Table 21). Neither context was used to train the decoder, and we find it interesting that this works. Both results are close to those of the trained voxel contexts in Table 5 (IoU 0.82–0.85, CD 1.67–2.00).
  • Conflicting inputs. When one input fully constrains the shape, it dominates: with 16316^3 voxels and a caption of a different object, all seeds follow the voxels. When both leave room, the model often combines them: a horse in a tall, narrow bounding box becomes, in two of three seeds, an upright horse that fits the box, and in the third a more unusual, inverted double-headed shape that also fills it (Fig. 7 in App. I of the revised paper).
  • Composition at inference. The conditional predictions for an image and a text prompt can be composed at inference, which gives simple text-driven edits of an image-conditioned shape without retraining (Fig. 8 in App. I of the revised paper).

(continued in part 2/2)

2. Effect of cross-modal masking on paired generation (W2, Q2)

Checkpoints in Table 6. The reviewer is right that the paper did not state this, and we appreciate the careful check. Every pre-trained encoder in Table 6 is the 300k-iteration checkpoint from the Table 7 probes, including the cross-modal encoder in the rows labelled "Ours-S". The fixed, uniform and cross-modal variants are therefore compared at the same pre-training budget. The 1M-iteration encoder is used only for the final models. Sec. 4.2 and the Table 6 caption now state this.

Paired-input generation. This test was an excellent suggestion, since it targets the motivation for cross-modal masking directly. We evaluate the decoders of Table 6, one per encoder, on both pairs the reviewer suggests, and on voxels alone, to measure the gain from the second modality. Following the reviewer's point on the 250 objects and the missing variance, results are the mean ± std over 3 sampling seeds on 1,250 Toys4K objects. Table 6 itself uses a smaller set of 250 objects, since it covers seven variants on five tasks each, so its numbers are not identical to the voxel-only rows below. We can re-run the rest of Table 6 on the 1,250 objects if the reviewer would like.

ContextEncoderCLIP-text ↑CLIP-image ↑IoU ↑CD ↓
voxelFixed0.2091 ± 0.00030.7076 ± 0.00010.7450 ± 0.00143.098 ± 0.026
Uniform0.2098 ± 0.00010.7079 ± 0.00050.7454 ± 0.00123.122 ± 0.037
Cross-modal0.2135 ± 0.00030.7151 ± 0.00040.7571 ± 0.00152.872 ± 0.012
voxel + textFixed0.2102 ± 0.00030.7131 ± 0.00010.7469 ± 0.00122.965 ± 0.018
Uniform0.2109 ± 0.00030.7140 ± 0.00020.7474 ± 0.00132.982 ± 0.018
Cross-modal0.2179 ± 0.00020.7255 ± 0.00030.7547 ± 0.00102.812 ± 0.010
voxel + imageFixed0.2150 ± 0.00010.7251 ± 0.00020.7496 ± 0.00232.975 ± 0.185
Uniform0.2158 ± 0.00010.7278 ± 0.00050.7508 ± 0.00042.812 ± 0.018
Cross-modal0.2227 ± 0.00030.7390 ± 0.00050.7781 ± 0.00032.452 ± 0.015

16316^3 voxels; 1,250 Toys4K objects; mean ± std over 3 sampling seeds; CD ×10³.

Cross-modal masking is best on every metric in all three settings. Adding an image helps the cross-modal encoder most: IoU improves by 0.021, against 0.005 for fixed and uniform masking, and Chamfer distance by 0.42, against 0.12 and 0.31, well above the variation across sampling seeds. Adding text raises both CLIP scores most for the cross-modal encoder (CLIP-text +0.0044, against +0.0011), but changes geometry little for all three, and the cross-modal IoU is slightly lower than with voxels alone (0.7547 vs. 0.7571). This is in line with the two inputs' roles, since the 16316^3 voxels already fix the coarse shape. The results are summarised in Sec. 4.2, with the full tables in App. H (Tables 19 and 20).

We hope these changes address the reviewer's concerns, and we are happy to provide further details.

Reviewer_XecMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

3Dify-Anything introduces a multi-conditional 3D geometry generator based on a "large encoder + small decoder" architecture. Various modalities (text, images, point clouds, voxels, and 3D shapes) are first discretized using their respective tokenizers and then combined into a shared vocabulary of approximately 154K tokens. The model employs a 407M BERT-style transformer trained from scratch. During pre-training, a cross-modal masking strategy is applied, utilizing 3D shape tokens as anchors to achieve implicit alignment across different modalities. Finally, these multi-modal conditions are used to generate 3D content via a DiT-base model, leveraging a rectified-flow formulation.

Critical strengths
  1. Ablation studies support the architectural choices. The design choice of allocating more capacity to the encoder while keeping the decoder lightweight is supported by two ablation studies: a parameter-matched decoder-only baseline and an encoder trained from scratch. The results demonstrate substantial improvements from the proposed design.

  2. A good balance between efficiency and quality. Image-to-3D generation is 3.3× faster than Trellis, with only a slight reduction in quality that remains within an acceptable range. Text-to-3D performance on Toys4K is largely on par with Trellis. Meanwhile, the method surpasses WaLa in IoU for both point cloud and voxel reconstruction.

  3. Extensive qualitative results suggest good generalization.

  4. The paper is easy to follow and provides thorough implementation details, including task specifications, tokenizer configurations for each modality, sampling settings for each modality, and rotation handling.

Critical weaknesses
  1. "Unified semantic space" appears somewhat overclaim. The unified vocabulary seems to standardize representation formats, but evidence of semantic alignment across modalities is insufficient.

  2. Important baselines are missing. Hunyuan3D-Omni is excluded due to its image-input requirement, despite comparable image+voxel, image+bbox, and image+point cloud settings. Recent unified methods, including Hunyuan3D-Buffalo and CG-MLLM, should also be discussed or evaluated.

  3. Qualitative comparisons are insufficient. The paper lacks visual comparisons with competing methods, particularly for image-to-3D and text-to-3D generation, where quantitative metrics alone cannot fully capture performance differences.

Questions for authors
  1. Ablation study scale and convergence trends. In the ablation study using 250 objects trained for 100k iterations, does the performance gap narrow as training progresses, potentially reaching a crossover or even surpassing the baseline? Why was the full Toys4K dataset not evaluated?

  2. Model behavior under conflicting conditions. When text conflicts with voxel or bounding box conditions, which condition does the model prioritize? For example, if the text prompt requests a bar stool but the bounding box corresponds to a sofa, which object is the model more likely to generate?

  3. Training costs and computational resources. How many training iterations and how much computational resources are required for the decoder? What computational resources are needed to train the 3D tokenizer and voxel tokenizer, respectively?

3 comments

We thank the reviewer for the detailed review. We are glad the reviewer found that the ablations support the encoder–decoder split, that the method balances speed and quality, and that the implementation details are thorough.

The reviewer's concerns are the "unified semantic space" claim, missing baselines and missing qualitative comparisons, with three questions on the ablation, conflicting conditions and training cost. We answer each below and in parts 2/3 and 3/3; details are in the revised paper (changes in red).

1. "Unified semantic space" (W1)

We thank the reviewer for this precise observation, which identifies a distinction that our wording blurred. A shared vocabulary unifies the format of the inputs, which is not the same as aligning their meaning: it only puts every modality into one token stream, and any alignment has to come from cross-modal pre-training.

What our results support is a narrower statement: the pre-trained encoder relates and combines modalities. Each pair of modalities clusters better than either of its parts (voxel + text AMI 0.728 vs. 0.631 and 0.531, Table 7), and in paired generation an added modality improves agreement even with the modality that is not given (with the cross-modal encoder, adding an image raises CLIP-text by 0.009 and adding text raises CLIP-image by 0.010, against 0.006 for fixed and uniform masking; Table 20). What they do not show is that features of different modalities share one embedding space.

We have therefore revised the wording to say exactly this: "a single uniform language" (Sec. 3.2) becomes "a single token vocabulary", and "better aligned, more semantic representations" (Sec. 1) becomes "representations that combine modalities better than standard masking does, as measured by zero-shot clustering". We also removed the remaining alignment wording: the encoder "must align the modalities with each other" (Sec. 1) becomes "must relate the modalities to each other", and "aligned cross-modal representations" (App. B) becomes "joint cross-modal representations".

2. Missing baselines (W2)

Hunyuan3D-Omni. Its image + control settings match three of our trained pairs, and its code and weights are public. We compare on the 1,250 Toys4K objects that TRELLIS uses for evaluation, with the same image and control for both models and Hunyuan3D-Omni's default settings. For the voxel control we follow its paper and pass the centres of the occupied cells of the 16316^3 grid.

InputModelIoU ↑CD ↓CLIP ↑
image + 16316^3 voxelsHunyuan3D-Omni0.45916.660.764
Ours-L0.8491.680.770
image + point cloudHunyuan3D-Omni0.8451.520.788
Ours-L0.9440.960.780
image + bounding boxHunyuan3D-Omni0.61411.170.789
Ours-L0.7117.460.764

One sample per object; CD ×10³; CLIP measures agreement with the input image. Best value per input in bold.

Ours-L reconstructs the geometry more closely in all three settings, while Hunyuan3D-Omni is slightly ahead on CLIP with a point cloud or a bounding box. On one H100, Ours-L takes about 1.5 s per object on average across the three settings, against 37.6 s for Hunyuan3D-Omni (about 25× faster). Hunyuan3D-Omni also always requires an image, whereas our model also generates from single inputs and from pairs without an image (App. J, Table 24).

(continued in part 2/3)

2. Missing baselines (W2), continued

CG-MLLM. We run its released model with default settings against Ours-L on the same objects, views and prompts:

ModelI→3D CLIP ↑I→3D Uni3D ↑I→3D ULIP2 ↑T→3D CLIP ↑T→3D Uni3D ↑T→3D ULIP2 ↑Time / object ↓
CG-MLLM (2B)0.7630.3140.3750.2350.2730.30532–36 s
Ours-L0.7680.3520.4030.2420.2940.3161.47 s

Ours-L scores higher on all six metrics and generates a shape more than 20× faster (App. J, Table 25).

Hunyuan3D-Buffalo. It appeared on arXiv in August 2026, and no inference code or weights have been released as of November 12, 2026, so we compare at the design level (App. J, Table 23). Like CG-MLLM, it takes text and image prompts and builds on a vision-language model, whereas we cover seven further input types with a 407M encoder and a small decoder.

3. Qualitative comparisons (W3)

Figs. 9 and 10 in App. J of the revised paper compare Ours-L with TRELLIS, WaLa, ShapeLLM-Omni and CG-MLLM on image-to-3D and text-to-3D. Ours-L follows the input image closely and keeps thin structures intact, such as butterfly wings, and it captures distinctive details of the prompt, such as the angular body of a guitar, where the baselines produce a standard guitar.

4. Ablation scale and convergence (Q1)

This is an important question, and we are grateful to the reviewer for raising it.

Evaluation set. Table 6 is evaluated on a smaller set of 250 Toys4K objects, since it covers seven variants on five tasks each. Our main Toys4K results use 1,250 objects, the standard protocol of TRELLIS, which keeps our numbers comparable with the baselines. Where we re-ran the ablation on the 1,250 objects with 3 sampling seeds (our response to Reviewer 26rv), the cross-modal encoder remains best on every metric. We can re-run the rest of Table 6 on the 1,250 objects if the reviewer would like. Sec. 4 now states both evaluation sets.

Convergence. All models in Table 6 are trained for 100k iterations, so we cannot show the trend beyond that point directly: training the seven variants to convergence would take well over 10k H100 GPU-hours. We expect the gap to narrow with longer training, but we do not expect a crossover, and the conclusions do not depend on one.

  • The flow loss alone is a weak teacher for the context. The same encoder trained jointly from scratch is barely better than no encoder at all (image ULIP2 0.250 vs. 0.244); pre-trained, it reaches 0.364.
  • Faster convergence is itself the benefit. At 100k iterations, about 2% of its schedule, Ours-S is already at image ULIP2 0.364, against 0.386 fully trained (different evaluation sets and encoder checkpoints, so indicative only), while the single transformer is at 0.267. A longer schedule would have to be paid for every generator, whereas our encoder is pre-trained once and shared by all three decoders.
  • The efficiency advantage does not depend on training length. The single transformer stays 1.8× slower, and even the 66M decoder-only model is slower than ours (875 vs. 654 ms).

Sec. 4.2 now states that the ablation uses a matched generator budget of 100k iterations, and the limitations (App. G) note that the baselines' behaviour beyond this budget is untested.

(continued in part 3/3)

5. Conflicting conditions (Q2)

We appreciate this question, and the reviewer's bar-stool example made a useful test. Training contains no conflicting pairs, so we tested this qualitatively with Ours-L, with three seeds each (Fig. 7 in App. I of the revised paper). Which input wins depends on how strongly the geometric input constrains the shape.

  • Voxels dominate. With voxels of a chair and a caption describing a bar stool, all three seeds follow the voxels.
  • A bounding box is combined with the text. For the reviewer's example, a bar stool in a wide, low box, the model produces neither a sofa nor a plain bar stool, but a wide, low frame built from stool-like legs and rungs that fills the box.

The model resolves such partial conflicts in more than one way: a horse in a tall, narrow box stands upright in two of three seeds, and in the third becomes a more unusual, inverted double-headed horse that also fills the box.

6. Training cost (Q3)

Thank you for asking; we agree that these details should be reported. All components are trained from scratch on nodes of 8 H100 GPUs (8 B200 for Ours-L). The context encoder is trained once and shared by all three decoders. This table is Table 28 in App. M.

ComponentGPUsBatchIterationsGPU-hours
Ours-S decoder (64.9M)8 H1002564.51M1.9k
Ours-M decoder (574.1M)8 H1002564.43M4.0k
Ours-L decoder (1.56B)8 B2002562.65M3.5k†
3D tokenizer (83×68^3 \times 6 FSQ)8 H100643.68M5.6k
Voxel tokenizer (16316^3 FSQ)8 H100643.00M0.6k
Context encoder (407M)32 H10010241.00M6.1k

GPU-hours = number of GPUs × logged training wall-clock, rounded. †B200 GPU-hours, roughly 7.0k H100-equivalent. Evaluation and ablation runs are not included.

We thank the reviewer again and are happy to discuss further.

Reviewer_srEiOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a unified model for conditional 3D shape generation from text, images, sketches, depth, point clouds, voxels and bounding boxes, etc. Modality-specific tokenizers (voxel, point cloud, image, and text tokenizer) feed a shared context encoder pretrained through masked prediction. The frozen encoder subsequently conditions a rectified-flow transformer on wavelet-based shape latents. Experiments demonstrate the effectiveness of the joint modeling, which supports broad conditioning support, competitive generation and reconstruction results.

Critical strengths
  • The model design is reasonable and complete. This work jointly models text, images, sketches, depth, point clouds, voxels and bounding boxes. As a result, the same model accommodates a broad range of semantic, visual, and geometric inputs while retaining competitive performance. This reduces the need for separate task-specific generators and provides a useful unified spacex` for controllable shape generation. Importantly, the voxel tokenizer and 3D Asset tokenizer demonstrate that different shape representations could also be unified.

  • The triaining strategy is effective. Context pretraining includes both conditioning modalities and target shape tokens. The pretrained encoder then provides conditioning for a lightweight flow decoder, separating representation learning from shape generation. The generation ablations provide useful evidence for the benefits of pretraining and the proposed masking strategy.

Unified modeling has shown promise for text and images, and this work extends that direction to 3D by bringing diverse semantic, visual, and geometric inputs into a single framework. The broad experimental evaluation and effective handling of different 3D representations make this a useful contribution to unified multimodal modeling, and I recommend acceptance.

Critical weaknesses

Here are some minor issues:

  • The relationship to recent unified 3D models is not sufficiently established. There are two recent works that may relate to unified modeling for 3D: UniVerse3D and Omni123. UniVerse3D also combines a multimodal understanding model with a 3D diffusion decoder, while Omni123 studies unified image to 3D generation through cross-modal training. A brief discussion of the differences would better establish the contribution.

  • Representation choice. Why use separate Cube and wavelet-based FSQ tokenizers for input point clouds and target shapes? What quality or efficiency advantages motivate this two seperate tokenizer choice over sharing the Cube tokenizer? A brief discussion will help to understand the motivation of the tokenizer design.

Questions for authors

I am also curious about the choice of training strategy. The context encoder is pretrained with token prediction and then frozen during flow training. For text-to-3D, have you tried to use a transfusion style shared transformer, applying next-token CE to text positions and flow matching to noisy 3D latent positions, instead of pretraining and freezing a separate context encoder? How would this joint training affect cross-modal transfer, 3D generation quality, and inference efficiency? This is a broader discussion and an explanation is enough.

3 comments

We thank the reviewer for the positive assessment and the recommendation to accept. We are glad the reviewer found the design reasonable and complete, with one model covering a broad range of semantic, visual and geometric inputs, and valued the separation of representation learning from shape generation.

The reviewer raises two points, on the relationship to recent unified 3D models and on the choice of tokenizers, and one broader question on the training strategy. We answer the first here and the other two in part 2/2; the full discussion is in the revised paper (changes in red).

1. Relationship to UniVerse3D and Omni123 (W1)

We thank the reviewer for pointing out these two works. We have added a short discussion to Sec. 2 and a full comparison to App. J (Table 22).

Each work unifies along a different axis. UniVerse3D unifies tasks: 3D understanding, generation and editing, around one vision-language model. Omni123 unifies output modalities: one autoregressive model generates both images and shapes, so that abundant 2D data can make up for scarce 3D data. We unify the conditioning: one generator accepts many visual, geometric and textual inputs, or pairs of them, and is designed to be fast. The three works are therefore complementary rather than competing. UniVerse3D and Omni123 broaden what a model can do or learn from, with text and images as inputs; we broaden what a 3D generator can be controlled by, and keep its inference cheap.

What we share. Like UniVerse3D, our model has two parts: one reads the inputs, and a separate generator conditioned on its hidden states produces the shape. Its ablation reaches the same conclusion as our Table 6, that the conditioning representation is what matters most. Like Omni123, we learn by predicting one modality's tokens from another's. In Omni123 this is the generation objective of the whole 2.2B model; in ours it is a masked pre-training objective for the encoder alone, which is trained once and then frozen.

What is different. Our design combines the two ideas: an encoder pre-trained with cross-modal prediction conditions a separate, lightweight generator. This combination is what gives its two main advantages.

  • A much broader range of conditions. Our model takes 9 single conditions (text, images, sketches, depth, masked images, point clouds, part point clouds, 16316^3 voxels and bounding boxes) and 13 pairs of them, 22 input formats in total (Table 14). UniVerse3D and Omni123 each generate from 2 single conditions, text and images, and from the two together (both also take a 3D shape for editing). This breadth comes from pre-training our encoder from scratch on a vocabulary that already contains the geometric inputs and the 3D shape tokens, so each input type is simply another token range in the same sequence.
  • Lightweight generation, made possible by encoder pre-training. Cross-modal pre-training makes a 407M encoder strong enough for a small decoder to rely on, and the encoder–decoder split means that this encoder runs only once (33 ms) while a 65M–1.56B decoder runs the multi-step diffusion. Together they give image-to-3D in 0.56–1.47 s on one H100. Table 6 shows that the pre-training is what makes this work: the same encoder trained from scratch with the decoder is far behind (image ULIP2 0.250 vs. 0.364). UniVerse3D instead reads the inputs with a Qwen2.5-VL model (at least 3B parameters) and generates with a 3.3B diffusion transformer, and Omni123 does both jobs in one 2.2B model that decodes 1,024 shape tokens one at a time.

(continued in part 2/2)

2. Why two shape tokenizers (W2)

We thank the reviewer for this question, and for the insight that the Cube tokenizer could in principle be shared. The paper did not explain the choice clearly.

Cube is used only to encode input point clouds and part point clouds. Our wavelet tokenizer cannot encode them: it takes a 2563256^3 TSDF of a complete mesh, which a point cloud or a part does not provide. For the target shapes, the reviewer is right that Cube could also be used. We use the wavelet tokenizer because it reconstructs shapes much more accurately, and the target tokenizer sets the quality ceiling of every generated shape. On the 1,250 Toys4K evaluation objects it more than halves the IoU error (0.963 vs. 0.913), lowers Chamfer distance by 23%, and is better on almost every object:

MetricOurs (wavelet, 83×68^3 \times 6)CubeOurs better on
IoU ↑0.9630.91399.9% of objects
Chamfer distance ↓2.112.7397.3% of objects

Reconstruction on the 1,250 Toys4K evaluation objects; IoU on the voxel grid; CD ×10³.

Efficiency is not the reason for the choice, since both tokenizers produce 512 tokens per shape. Using two tokenizers also costs little, because predicting 3D shape tokens from a point cloud is one of the cross-modal pre-training tasks. This is now explained in App. K (Table 26), with visual examples in Fig. 11 of the revised paper.

3. Transfusion-style joint training instead of a frozen encoder (Q)

We thank the reviewer for this interesting question. We have not trained a Transfusion-style model, but Table 6 contains two closely related setups trained with the flow loss only: a single transformer that reads the context and generates the shape, and our encoder trained jointly with the decoder from scratch. They match Transfusion's shared design, but not its next-token loss on text.

  • Generation quality. The frozen pre-trained encoder is better than the single transformer on 12 of 13 metrics, and better than joint training on all 13. The gap is largest for image, depth and point-cloud inputs (e.g. image ULIP2 0.364 vs. 0.267). A next-token loss on text would most likely help text, but it does not target these inputs.
  • Inference. The next-token loss itself adds no cost when generating a shape, but the shared design does: the 512 latent positions pass through the whole model at every solver evaluation, whereas our encoder runs once and only the small decoder iterates. At matched parameters, the single transformer is 1.8× slower (1197 vs. 654 ms). If the model were also asked to generate text, as Transfusion can, that text would be decoded one token at a time.
  • Cross-modal transfer. A next-token loss on text transfers to 3D only indirectly, through shared weights, and it is causal, so the order of modalities matters. Our masked objective predicts each modality from the other, in both orders, with bidirectional attention.

The full discussion and numbers are in App. L (Table 27).

We thank the reviewer again and are happy to discuss further.

I have read the authors' response and have no further concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.