- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
Omni123 studies how to improve 3D generation when 3D training data is limited, by exploiting large amounts of 2D image data. The authors propose a unified autoregressive model that represents text, images, and 3D shapes as discrete 1D tokens and trains them jointly within a shared multimodal framework. In what follows, I outline the strengths of the paper, its weaknesses, and the questions I would like the authors to respond. If these concerns are properly addressed, I would be willing to increase my score.
Critical strengths
1.The paper tackles the scarcity of 3D training data and attempts to leverage abundant 2D data to improve 3D generation. The research problem is meaningful and well-motivated.
2.The proposed 3DEdit-1M dataset is large in scale. For Consis3D-Bench, target shapes are hand-modeled by artists and the unedited regions are strictly preserved, which enables a more reliable evaluation of consistency in 3D editing.
3.Omni123 achieves leading results on the main metrics of Consis3D-Bench. In addition, Table 7 shows that Joint PT outperforms Interleaved PT; this finding provides a valuable empirical takeaway for future work.
Critical weaknesses
1.Why is an autoregressive model chosen over diffusion, which is the dominant paradigm for 3D generation? A plausible reason is that the generated image is meant to serve as the condition for 3D generation (since the image tokens are concatenated before the 3D tokens). If that is the case, I would argue that the main contribution of this work lies in generating images in a "Blender-rendered" style that is more favorable to the 3D generator — something that could probably be achieved simply by fine-tuning an image generator — rather than in a complex unified framework.
2.Lack of image-to-3D evaluation. Image to 3D accounts for the largest share of the total pretraining tokens, yet it has no results in the main paper; 3D to image is likewise not evaluated.
3.Insufficient discussion and comparison with recent methods. The paper does not quantify the capability gap between current unified methods and specialized methods. The main results are mostly compared against Cube3D, ShapeLLM-Omni, and earlier cascaded pipelines. At minimum, the following should be discussed and compared (if possible):
Specialized:
TRELLIS.2 (CVPR 2026),
SAM 3D Objects (CVPR 2026),
Unified:
CG-MLLM (ICML 2026).
Questions for authors
1.In Fig. 8, the output of Sana + Hunyuan3D-2.1 looks visually better than that of ShapeLLM-Omni, whereas the ULIP2-T / Uni3D-T ranking in Table 4 is reversed. Could the authors explain this inconsistency? Or complement the reported metrics with a user study so that their trustworthiness can be properly assessed?
2.Given causal attention and the token order (image tokens before 3D tokens), does Omni123 essentially reduce to conditional 3D generation from text + image? Starting from the start-of-3D token, the model is effectively generating 3D conditioned on the preceding text and image tokens. In that case, what would happen if one directly feeds a text+GT-Render image into the framework, i.e., uses it as a plain text+image-to-3D model? Would such a simplified setup match or even outperform the full unified pipeline?
3 comments
I appreciate the authors' response, which has addressed most of my concerns. I will raising my score from 2 to 3, conditional on the inclusion of the early-stage architectural experiments and comparisons with more recent methods in the revised version. These designs and experiments would provide readers with a more comprehensive understanding of the proposed approach.
Thank you for raising your score and for your thoughtful consideration of our response. We will incorporate all your suggestions into the final version to provide a clearer and more detailed presentation of our approach.
Rebuttal of Omni123 for Reviewer Wvvd
1. Choice of architectural design (W1)
We aim to build a unified multimodal model and extend its capabilities to native 3D editing, exploring a path toward 3D Nano-Banana, rather than to improve image-to-3D generation. We therefore adopt autoregressive, a widely adopted and well-established paradigm in multimodal learning. We transfer knowledge from general-domain image data(L77–79) to 3D, with image generation serving as constrain between 2D observations and 3D shapes. This further motivates our choice to pretrain for better modality unification.As evidenced in previous multimodal models (VILA-U, InfinityStar, JavisDiT++, MMAudio), single-modal experts have already overfit to the specific domain. Fine-tuning them for unified multimodal generation is more difficult and less efficient than pretraining a unified model. In our early attempts, we built an additional 3D diffusion head on top of Qwen-Image and Qwen2.5-VL-7B, but observed very poor text–3D and generated image–3D alignment (Table 1), indicating ineffective cross-modal unification and 2D-to-3D knowledge transfer.
Table 1.
Gen measures alignment between generated images and 3D shapes.
2. Evaluation of cross-modal tasks (W2)
We report image-to-3D and mv-to-3D results in Supp.C.3, Tab.S7. We evaluate 3D-to-image on the NVS in Tab.S7(d). We will improve the layout in our final revision for better clarity.
3. More Baselines (W3)
Thank you for pointing this out. We add comparisons with TRELLIS.2, SAM 3D Objects and CG-MLLM in Table 2. For image-to-3D models, we follow the setup in Tab.4 and use Sana(1.6B) to generate images. We initially didn't compare them because of parameter or architecture differences. Only TRELLIS.2 outperform us since it is a image-to-3D expert model with 4B parameters, while Omni123's 2B parameters need to be shared across all multimodal tasks.
Table 2. More Baseline Comparisons.
4. Visual quality and text–3D alignment (Q1)
Thank you for your feedback. The metrics in Tab.4 measure text–3D alignment, which is a different aspect from visual quality shown in Fig.8. Here in Table 3, we provide the complete prompts for the examples in Fig.8, revealing semantic misalignments caused by error accumulation in the cascaded pipeline. Also as shown in Fig.8, Sana + Hunyuan3D-2.1 exhibits severe issues in detailed geometry and part composition (it missed the wizard’s staff and the robot’s left hand).
Table 3. Prompts for Fig.8
5. Image-generation supervision and conditional 3D inference (Q2)
Thank you for your insight and interest. Our purpose in incorporating tasks related to image generation as supervision is to make image generation a bridge between text and 3D, transferring knowledge from abundant 2D observations to strengthen 3D representations. General-domain text-to-image training brings broader concepts, appearances, and spatial relationships to 3D learning. Training solely on text+image-to-3D does not address the underlying limitations of available 3D data. As shown in the scaling curves in Fig.3, using only 3D data encounters a bottleneck due to limited accessible 3D assets, whereas joint training continuously benefit from vast amounts of image data.
At inference, Omni123 can generate 3D conditioned on externally provided images. In Table 4, we use Qwen-Image-2.1 to generate images for each prompt in the test set, then feed both the text and image into Omni123.
Table 4. Comparison of Inference Setup