- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
3Dify-Anything is a single conditional 3D generator for visual, geometric and textual inputs. A context encoder, pretrained with masking, processes the conditioning once, and a rectified-flow decoder then generates a wavelet-based 3D latent. The proposed pretraining keeps one modality of a pair fully visible and masks the other, so that the encoder learns to predict across modalities. The paper evaluates several single-input tasks and some paired inputs, and ablates the pretraining, the masking objective and the decoder size.
Critical strengths
-
The paper gives good evidence that the limiting factor is the context representation rather than decoder capacity, which is the core argument for its design. In Table 6, the pretrained model clearly beats a parameter-matched decoder-only model on image, depth and point-cloud conditioning (voxel and text are roughly on par), and training the same encoder from scratch together with the decoder helps little. Table 7 shows that cross-modal masking clearly improves paired-input probes (e.g. image+text AMI 0.355 → 0.592), and Table 16 shows little change from Ours-M to Ours-L, although going from S to M still helps. This is the main reason I lean towards acceptance: it justifies putting capacity into a pretrained encoder instead of a larger decoder.
-
The design is practical and the results are strong. The large encoder runs once and only a light decoder iterates, which gives about 0.56 s (Ours-S) and 1.47 s (Ours-L) for image-to-3D on one H100 (Tables 1 and 17). On geometric inputs, all three model sizes beat the listed baselines on point-cloud-to-3D IoU on both MAS and GSO (Table 3), and Ours-M/L clearly improve voxel-to-3D IoU over WaLa at the same resolution (Table 4). Fast generation from many input types with one model is useful in practice.
Critical weaknesses
-
The "any combination" claim is broader than what is shown. The decoder is trained on 22 tasks, all single inputs or pairs (Table 14). Table 5 gives numbers only for voxel, voxel+text and voxel+image, the other trained pairs appear only in figures, and as far as I can tell every combination shown is one of the pairs in Table 14. So the paper supports a unified model for a set of trained input formats, but not generalization to unseen combinations or to more than two inputs. This affects how the contribution is stated more than the method itself, but since it is the headline claim, it should either be tested or narrowed.
-
The masking objective is the main technical novelty, but its effect on generation from paired inputs, which is its motivation, is not tested. All generation tasks in Table 6 use a single conditioning modality. There, cross-modal masking beats uniform masking on four of the five tasks by modest margins (e.g. image ULIP2 0.340 → 0.364), point cloud slightly favors uniform (IoU 0.914 vs 0.910), and with 250 objects and no variance these gaps are hard to read. Paired inputs are evaluated only through clustering (Table 7), and paired generation (Table 5) only for the final model. It is also not stated which pretraining checkpoint the fixed and uniform encoders in Table 6 use: the probes compare encoders at 300k iterations, while the "ours" row is labeled Ours-S, whose encoder is pretrained for 1M iterations (Sec. 4). This is what keeps my score at marginal acceptance. A paired-input generation comparison of the masking variants would address most of my concern.
Questions for authors
-
Is the intended claim support for the trained single/paired formats, or generalization to combinations not seen in decoder training? If the latter, can the authors evaluate one held-out combination?
-
Can the fixed/uniform/cross-modal encoders be compared on a paired generation task (voxel+image or voxel+text) with the same decoder budget? Which pretraining checkpoint does each Table 6 row use?
2 comments
2. Effect of cross-modal masking on paired generation (W2, Q2)
Checkpoints in Table 6. The reviewer is right that the paper did not state this, and we appreciate the careful check. Every pre-trained encoder in Table 6 is the 300k-iteration checkpoint from the Table 7 probes, including the cross-modal encoder in the rows labelled "Ours-S". The fixed, uniform and cross-modal variants are therefore compared at the same pre-training budget. The 1M-iteration encoder is used only for the final models. Sec. 4.2 and the Table 6 caption now state this.
Paired-input generation. This test was an excellent suggestion, since it targets the motivation for cross-modal masking directly. We evaluate the decoders of Table 6, one per encoder, on both pairs the reviewer suggests, and on voxels alone, to measure the gain from the second modality. Following the reviewer's point on the 250 objects and the missing variance, results are the mean ± std over 3 sampling seeds on 1,250 Toys4K objects. Table 6 itself uses a smaller set of 250 objects, since it covers seven variants on five tasks each, so its numbers are not identical to the voxel-only rows below. We can re-run the rest of Table 6 on the 1,250 objects if the reviewer would like.
| Context | Encoder | CLIP-text ↑ | CLIP-image ↑ | IoU ↑ | CD ↓ |
|---|---|---|---|---|---|
| voxel | Fixed | 0.2091 ± 0.0003 | 0.7076 ± 0.0001 | 0.7450 ± 0.0014 | 3.098 ± 0.026 |
| Uniform | 0.2098 ± 0.0001 | 0.7079 ± 0.0005 | 0.7454 ± 0.0012 | 3.122 ± 0.037 | |
| Cross-modal | 0.2135 ± 0.0003 | 0.7151 ± 0.0004 | 0.7571 ± 0.0015 | 2.872 ± 0.012 | |
| voxel + text | Fixed | 0.2102 ± 0.0003 | 0.7131 ± 0.0001 | 0.7469 ± 0.0012 | 2.965 ± 0.018 |
| Uniform | 0.2109 ± 0.0003 | 0.7140 ± 0.0002 | 0.7474 ± 0.0013 | 2.982 ± 0.018 | |
| Cross-modal | 0.2179 ± 0.0002 | 0.7255 ± 0.0003 | 0.7547 ± 0.0010 | 2.812 ± 0.010 | |
| voxel + image | Fixed | 0.2150 ± 0.0001 | 0.7251 ± 0.0002 | 0.7496 ± 0.0023 | 2.975 ± 0.185 |
| Uniform | 0.2158 ± 0.0001 | 0.7278 ± 0.0005 | 0.7508 ± 0.0004 | 2.812 ± 0.018 | |
| Cross-modal | 0.2227 ± 0.0003 | 0.7390 ± 0.0005 | 0.7781 ± 0.0003 | 2.452 ± 0.015 |
voxels; 1,250 Toys4K objects; mean ± std over 3 sampling seeds; CD ×10³.
Cross-modal masking is best on every metric in all three settings. Adding an image helps the cross-modal encoder most: IoU improves by 0.021, against 0.005 for fixed and uniform masking, and Chamfer distance by 0.42, against 0.12 and 0.31, well above the variation across sampling seeds. Adding text raises both CLIP scores most for the cross-modal encoder (CLIP-text +0.0044, against +0.0011), but changes geometry little for all three, and the cross-modal IoU is slightly lower than with voxels alone (0.7547 vs. 0.7571). This is in line with the two inputs' roles, since the voxels already fix the coarse shape. The results are summarised in Sec. 4.2, with the full tables in App. H (Tables 19 and 20).
We hope these changes address the reviewer's concerns, and we are happy to provide further details.
We thank the reviewer for the thoughtful feedback. We are glad the reviewer found the paper's core argument well supported, namely that the limiting factor is the context representation rather than decoder capacity, and found the design practical, with fast generation and strong results on geometric inputs.
The reviewer's concerns are the scope of the "any combination" claim and the effect of cross-modal masking on paired-input generation. We answer the first here and the second in part 2/2; the corresponding changes are in the revised paper (in red).
1. Scope of the "any combination" claim (W1, Q1)
We thank the reviewer for this question, which helped us state the contribution more precisely. Our intended claim is that one model supports the trained input formats, not that it generalises to combinations unseen in training. The reviewer is right that the wording said more. The decoder is trained on 22 formats (9 single inputs and 13 pairs, Table 14), and every combination shown in the paper is one of them. We have narrowed the wording everywhere it appeared: the abstract ("or any combination of these" becomes "or one of 13 supported pairs of these"), the Fig. 1 caption, Secs. 1, 2, 4.1 and 5, and App. B. App. G has a new limitation stating that generalisation to unseen combinations, and to more than two inputs, is not established.
Beyond the trained formats. This is not part of our claim, but the reviewer may find three observations interesting (App. I).
(continued in part 2/2)