acceptodds
Under review as a conference paper at ICLR 2027

Omni123: Exploring 3D Native Foundation Models with Limited 3D Data by Unifying Text to 2D and 3D Generation

Abstract

Recent multimodal large language models have achieved remarkable success in unified text and image understanding and generation, yet extending such native capability to 3D remains an open challenge. The core bottleneck is data: compared to the near-infinite 2D imagery on the web, high-quality 3D assets are orders of magnitude scarcer, leaving 3D synthesis severely under-constrained. Current methods often circumvent this limitation through indirect pipelines that edit in 2D image space and lift results into 3D via iterative optimization, sacrificing geometric consistency and the straightforwardness of native generation. We present Omni123, a 3D native foundation model that addresses limited 3D data by unifying text-to-2D and text-to-3D generation within a single autoregressive framework. Our key insight is that cross-modal generative consistency between images and 3D can act as an implicit structural constraint: by representing text, images, and 3D geometry as discrete tokens in a shared sequence space, the model can leverage abundant 2D observations as a rich geometric prior to fortify 3D representations. To realize this, we introduce an interleaved X-to-X training paradigm that coordinates diverse cross-modal tasks over heterogeneous paired datasets without requiring fully aligned text–image–3D triplets. By traversing semantic–visual–geometric cycles (e.g., text → image → 3D → image) within single autoregressive sequences, the model learns representations that simultaneously satisfy high-level semantic intent, appearance fidelity, and multi-view geometric consistency, while mitigating harmful interference between appearance and geometry objectives. Extensive experiments demonstrate that Omni123 significantly improves geometric consistency and semantic alignment in text-guided 3D generation and editing, validating that unifying 2D and 3D generative processes provides an effective and scalable pathway toward multimodal 3D world models.

Primary area:
generative models
Keywords:
3D generation, 3D edit, unified model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WvvdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

Omni123 studies how to improve 3D generation when 3D training data is limited, by exploiting large amounts of 2D image data. The authors propose a unified autoregressive model that represents text, images, and 3D shapes as discrete 1D tokens and trains them jointly within a shared multimodal framework. In what follows, I outline the strengths of the paper, its weaknesses, and the questions I would like the authors to respond. If these concerns are properly addressed, I would be willing to increase my score.

Critical strengths

1.The paper tackles the scarcity of 3D training data and attempts to leverage abundant 2D data to improve 3D generation. The research problem is meaningful and well-motivated.

2.The proposed 3DEdit-1M dataset is large in scale. For Consis3D-Bench, target shapes are hand-modeled by artists and the unedited regions are strictly preserved, which enables a more reliable evaluation of consistency in 3D editing.

3.Omni123 achieves leading results on the main metrics of Consis3D-Bench. In addition, Table 7 shows that Joint PT outperforms Interleaved PT; this finding provides a valuable empirical takeaway for future work.

Critical weaknesses

1.Why is an autoregressive model chosen over diffusion, which is the dominant paradigm for 3D generation? A plausible reason is that the generated image is meant to serve as the condition for 3D generation (since the image tokens are concatenated before the 3D tokens). If that is the case, I would argue that the main contribution of this work lies in generating images in a "Blender-rendered" style that is more favorable to the 3D generator — something that could probably be achieved simply by fine-tuning an image generator — rather than in a complex unified framework.

2.Lack of image-to-3D evaluation. Image to 3D accounts for the largest share of the total pretraining tokens, yet it has no results in the main paper; 3D to image is likewise not evaluated.

3.Insufficient discussion and comparison with recent methods. The paper does not quantify the capability gap between current unified methods and specialized methods. The main results are mostly compared against Cube3D, ShapeLLM-Omni, and earlier cascaded pipelines. At minimum, the following should be discussed and compared (if possible):

Specialized:

TRELLIS.2 (CVPR 2026),

SAM 3D Objects (CVPR 2026),

Unified:

CG-MLLM (ICML 2026).

Questions for authors

1.In Fig. 8, the output of Sana + Hunyuan3D-2.1 looks visually better than that of ShapeLLM-Omni, whereas the ULIP2-T / Uni3D-T ranking in Table 4 is reversed. Could the authors explain this inconsistency? Or complement the reported metrics with a user study so that their trustworthiness can be properly assessed?

2.Given causal attention and the token order (image tokens before 3D tokens), does Omni123 essentially reduce to conditional 3D generation from text + image? Starting from the start-of-3D token, the model is effectively generating 3D conditioned on the preceding text and image tokens. In that case, what would happen if one directly feeds a text+GT-Render image into the framework, i.e., uses it as a plain text+image-to-3D model? Would such a simplified setup match or even outperform the full unified pipeline?

3 comments
AuthorsOpenReview ↗

Rebuttal of Omni123 for Reviewer Wvvd

1. Choice of architectural design (W1)

We aim to build a unified multimodal model and extend its capabilities to native 3D editing, exploring a path toward 3D Nano-Banana, rather than to improve image-to-3D generation. We therefore adopt autoregressive, a widely adopted and well-established paradigm in multimodal learning. We transfer knowledge from general-domain image data(L77–79) to 3D, with image generation serving as constrain between 2D observations and 3D shapes. This further motivates our choice to pretrain for better modality unification.As evidenced in previous multimodal models (VILA-U, InfinityStar, JavisDiT++, MMAudio), single-modal experts have already overfit to the specific domain. Fine-tuning them for unified multimodal generation is more difficult and less efficient than pretraining a unified model. In our early attempts, we built an additional 3D diffusion head on top of Qwen-Image and Qwen2.5-VL-7B, but observed very poor text–3D and generated image–3D alignment (Table 1), indicating ineffective cross-modal unification and 2D-to-3D knowledge transfer.

Table 1.

MethodUni3D-T↑ULIP2-T↑Uni3D-I(Gen)↑ULIP2-I(Gen)↑
Qwen-Image-2511+3D Gen0.26390.16550.24060.2547
Qwen2.5-VL-7B+3D Gen0.25740.1580--
Omni1230.29080.18290.26930.2805

Gen measures alignment between generated images and 3D shapes.

2. Evaluation of cross-modal tasks (W2)

We report image-to-3D and mv-to-3D results in Supp.C.3, Tab.S7. We evaluate 3D-to-image on the NVS in Tab.S7(d). We will improve the layout in our final revision for better clarity.

3. More Baselines (W3)

Thank you for pointing this out. We add comparisons with TRELLIS.2, SAM 3D Objects and CG-MLLM in Table 2. For image-to-3D models, we follow the setup in Tab.4 and use Sana(1.6B) to generate images. We initially didn't compare them because of parameter or architecture differences. Only TRELLIS.2 outperform us since it is a image-to-3D expert model with 4B parameters, while Omni123's 2B parameters need to be shared across all multimodal tasks.

Table 2. More Baseline Comparisons.

MethodParam.ULIP2-T↑Uni3D-T↑
Two-stage:Text→Image→3D
Sana+TRELLIS.21.6B+4B0.20360.3117
Sana+SAM3D Objects1.6B+1.8B0.17930.2875
Native:Text →3D
CG-MLLM3.14B0.14800.2453
ShapeLLM-Omni7B0.16490.2666
Ours2.2B0.18290.2908

4. Visual quality and text–3D alignment (Q1)

Thank you for your feedback. The metrics in Tab.4 measure text–3D alignment, which is a different aspect from visual quality shown in Fig.8. Here in Table 3, we provide the complete prompts for the examples in Fig.8, revealing semantic misalignments caused by error accumulation in the cascaded pipeline. Also as shown in Fig.8, Sana + Hunyuan3D-2.1 exhibits severe issues in detailed geometry and part composition (it missed the wizard’s staff and the robot’s left hand).

Table 3. Prompts for Fig.8

ExamplePrompt
Wizard statueA full-body statue of a standing elderly wizard, wearing a tall pointed hat. His long robe hangs smooth and flat, secured at the waist by a knotted belt. He holds a slender staff topped with three pointing prongs, while his other hand extends gently from a wide sleeve.
Robot in chairA slender, human-proportioned robot sits upright in a simple armchair, with its hands resting on its thighs and both feet flat on the floor. Its rounded oval head has two small, closely spaced eyes and circular ear discs. A narrow neck connects to a broad, curved chest plate that tapers toward an exposed mechanical waist. Rounded joints link its long, segmented arms and articulated fingers. Its cylindrical shins extend down to rounded feet.

5. Image-generation supervision and conditional 3D inference (Q2)

Thank you for your insight and interest. Our purpose in incorporating tasks related to image generation as supervision is to make image generation a bridge between text and 3D, transferring knowledge from abundant 2D observations to strengthen 3D representations. General-domain text-to-image training brings broader concepts, appearances, and spatial relationships to 3D learning. Training solely on text+image-to-3D does not address the underlying limitations of available 3D data. As shown in the scaling curves in Fig.3, using only 3D data encounters a bottleneck due to limited accessible 3D assets, whereas joint training continuously benefit from vast amounts of image data.

At inference, Omni123 can generate 3D conditioned on externally provided images. In Table 4, we use Qwen-Image-2.1 to generate images for each prompt in the test set, then feed both the text and image into Omni123.

Table 4. Comparison of Inference Setup

InferenceImage sourceULIP2-T↑Uni3D-T↑
Text+Image→3DQwen-Image-2.10.18340.2881
Text→Image→3DOmni1230.18290.2908

I appreciate the authors' response, which has addressed most of my concerns. I will raising my score from 2 to 3, conditional on the inclusion of the early-stage architectural experiments and comparisons with more recent methods in the revised version. These designs and experiments would provide readers with a more comprehensive understanding of the proposed approach.

Thank you for raising your score and for your thoughtful consideration of our response. We will incorporate all your suggestions into the final version to provide a clearer and more detailed presentation of our approach.

Reviewer_Ruf3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Omni123, a unified autoregressive framework for text-to-image and text-to-3D generation. By representing images and 3D shapes as discrete tokens and introducing cross-modal training, the method leverages abundant 2D data to improve 3D generation under limited 3D supervision. The authors further introduce 3DEdit-1M, a large-scale 3D editing dataset, and Consis3D-Bench for evaluating editing consistency. Experiments demonstrate improved geometric consistency and semantic alignment in 3D generation and instruction-based editing.

Critical strengths
  1. Promising Research Direction. Unifying 2D and 3D generation to leverage abundant 2D supervision offers a promising approach to addressing 3D data scarcity.
  2. Strong Experimental Results. Omni123 achieves competitive performance in text-to-3D generation and substantially improves instruction-based 3D editing, particularly in preserving non-edited geometry.
  3. Valuable Dataset and Benchmark. The proposed 3DEdit-1M dataset and artist-modeled Consis3D-Bench provide useful resources for advancing and evaluating instruction-based 3D editing.
Critical weaknesses
  1. Insufficient Ablation of Cross-Modal Knowledge Transfer. Table 7 shows that interleaved X-to-X pretraining actually underperforms standard joint multi-task pretraining. The main improvements emerge after multi-view cycle SFT, making it unclear whether the gains primarily originate from unified 2D–3D learning or additional multi-view supervision.
  2. Potential Bias in 3D Editing Evaluation. The proposed Consis3D-Bench uses source shapes and instructions drawn from the same synthetic data pipeline as 3DEdit-1M. Although target shapes are artist-modeled, this may favor Omni123 and raises questions about generalization to independent, real-world 3D editing tasks.
  3. Insufficiently Controlled Baseline Comparisons. The compared methods differ in model sizes, training data, and potentially computational budgets. Without controlled comparisons using comparable training data and compute, it is hard to isolate the benefits of the proposed training framework.
Questions for authors
  1. Can the model generalize to arbitrary camera viewpoints beyond the six predefined canonical views?
  2. Since the editing model takes both source images and 3D shapes as inputs, can it perform 3D editing using only the source 3D shape and text instructions?
3 comments
AuthorsOpenReview ↗

Rebuttal of Omni123 for Reviewer Ruf3

1. Cross-modal knowledge transfer (W1)

We clarify that our final model adopts joint multi-task pretraining, with interleaved cycle sequences introduced during SFT. The term “interleaved X-to-X” describes our final model’s ability to generate image and 3D content within a shared multimodal context, using previously generated content to condition subsequent generation, as described in previous interleaved models (OpenING, Chameleon, SenseNova-U1).

We adopt joint multi-task PT because it significantly outperforms interleaved PT on both ULIP2-T and Uni3D-T, as reported in Table 7(a–b) of the main paper. To further evaluate the impact of the first-stage training strategy on subsequent performance, we apply the same interleaved SFT settings to both pretrained models, as shown in Table 1.

Table 1. Comparison of pretraining and SFT strategies.

PretrainingSFTULIP2-T ↑Uni3D-T ↑
Interleaved X-to-X PTInterleaved multi-view cycle SFT0.17320.2754
Joint multi-task PT (Ours)Interleaved multi-view cycle SFT0.18290.2908

2. Potential bias in 3D editing evaluation (W2)

Our 3D editing comparisons in both the main paper (Section 5.2, Table 5) and supplementary material (Section C.6, Table S10) cover three benchmarks: Consis3D-Bench, Edit3D-Bench, and 3DEditVerse. Omni123 achieves state-of-the-art zero-shot performance on both external benchmarks, without training on Edit3D-Bench or the 3DEditVerse training set. These results provide evidence of generalization beyond our own data construction pipeline.

3. Controlled baseline comparisons (W3)

Modern large generative models rely on carefully designed multi-stage training pipelines and large-scale training data. Their complete pretraining details and scripts are not uniformly publicly available. Reproducing the full training pipeline of every baseline under matched data and compute would therefore incur unaffordable time and computational costs. Following the evaluation practice of prior works, including ShapeLLM-Omni, TRELLIS.2, Hunyuan3D-, and Sparc3D, we compare against existing pretrained models and select models with comparable parameter counts wherever possible (Section 5.1, Table 4).

To isolate the benefits of our training framework, we conduct task composition ablations during pretraining, as reported in Supplementary Section C.2, Table S6 (Lines 902–917). We train different Omni123 versions with identical total training tokens, compute budgets, and architectures, varying only the pretraining task composition: full joint training, joint training with each of these tasks individually removed, and single-task training. These ablations provide direct evidence that joint pretraining supplies complementary supervision and enables cross-modal knowledge transfer, demonstrating the effectiveness of our framework.

4. Generalization to arbitrary camera viewpoints (Q1)

Thanks for your interest. We did not originally treat NVS or multi-view-to-3D as primary tasks, but used them mainly as supervision to strengthen 2D–3D alignment during SFT. We therefore follow the choice of six views in prior works such as CRM [1] and Wonder3D [2] to ensure basic coverage of the object. We leave more flexible camera control to future work.

5. 3D editing without source images (Q2)

Thanks for your suggestion. Our model supports arbitrary input and output sequences, including 3D editing using only the source mesh and text instructions. In Table 2, we compare editing performance with and without the source image input.

Table 2. Ablation of source image input in 3D editing.

MethodCons.Dist.↓ULIP2↑DINO-I↑CLIP-I↑SSIM↑LPIPS↓
Edit3D-Bench
w/o source image—0.78360.75920.92080.89570.1339
default—0.78530.76440.92450.89460.1346
3DEditVerse
w/o source image—0.82510.78340.93370.91560.1033
default—0.82390.78650.93300.91770.1060
Consis3D-Bench
w/o source image0.01380.92530.88020.95370.93180.0574
default0.01340.92720.88590.95950.92960.0583

Removing the source image has only a minor impact, since the input mesh already provides sufficiently rich information for geometric editing.

References

[1] Zhengyi Wang et al. CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model. ECCV, 2024.

[2] Xiaoxiao Long et al. Wonder3D: Single Image to 3D using Cross-Domain Diffusion. CVPR, 2024.

Reviewer_Ruf3OpenReview ↗

Response

Thank you for the detailed rebuttal and additional experiments. Most of my concerns have been adequately addressed, particularly regarding the 3D editing generalization and controlled ablations.

My main remaining concern is the attribution of performance improvements to the proposed interleaved multi-view cycle. Although the authors clarified that the SFT configurations use matched training-token budgets and input views, it remains unclear whether the improvements arise specifically from interleaved sequence modeling or from multi-view supervision. A more controlled comparison would help establish this contribution.

Moreover, the new comparisons suggest that specialized pipelines can still outperform Omni123 in text-to-3D generation. I encourage the authors to clarify this distinction and moderate claims regarding the advantages of unified generation.

Overall, I appreciate the authors' efforts and find the work promising. I will maintain my rating of 3 (Weak Accept).

Thank you for your suggestions and understanding.

1. Contribution of multi-view SFT

Indeed, multi-view SFT brings clear improvements. However, Tables 4 and 7 show that Omni123 already outperforms many baselines with joint multi-task pretraining alone, before introducing multi-view interleaved cycle training, highlighting the effectiveness of joint pretraining. Moreover, Table 7 compares the “Single-view SFT” and “Multi-view cycle” configurations under matched input-view coverage and training-token budgets, providing a controlled evaluation of the additional cycle objective. In another aspect, the main motivation of using interleaved training is to strengthen cross-modal interactions and provide a foundation for downstream editing. This motivation is consistent with findings from unified multimodal models, for example BAGEL and SenseNova-U1

2. Comparison with specialized pipelines

Table 4 and supplementary Table S12 show that Omni123 outperforms most evaluated cascaded pipelines on the reported text-to-3D metrics. We acknowledge the remaining gap relative to TRELLIS.2. However, TRELLIS.2 itself is a specialized 4B-parameter image-to-3D model, and using it for cascaded text-to-3D generation requires an additional large text-to-image model, while Omni123 needs to share a single 2.2B-parameter backbone across multiple tasks.

Our primary effort focuses on exploring unified 3D multimodal modeling and effective cross-modal knowledge transfer, which already quite non-trivial and costs lots of computation. We leave further model scaling to future work.

Reviewer_gowKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Omni123 unifies text-conditioned image and 3D shape generation within an autoregressive framework using discrete image and geometry tokens. Its training pipeline combines joint multi-task pre-training, view-conditioned continued training, and interleaved multi-view cycle fine-tuning. The paper also introduces a two-stage image tokenizer, the synthetic 3DEdit-1M training dataset, and the artist-created Consis3D-Bench. Evaluation covers text-to-3D generation, instruction-based 3D editing, image reconstruction, and training-strategy ablations.

Critical strengths

1.The unified formulation provides a practical way to combine heterogeneous text–image, image–3D, and text–3D supervision. Table 7 provides evidence that multi-view cycle fine-tuning improves text–shape alignment over the single-view configuration from the same pre-training starting point. This supports the usefulness of coupling image and geometry prediction, while leaving room for more precise attribution of the improvement.

2.The editing evaluation addresses both instruction fidelity and preservation of unchanged geometry. Consis3D-Bench uses artist-created targets to enable a more direct assessment of preservation than independently reconstructed synthetic pairs. Table 5 reports strong results on this benchmark and competitive transfer to two external editing benchmarks, providing evidence beyond the newly constructed evaluation set.

Critical weaknesses

1.The contribution of cycle-based sequence organization is only partially isolated. Section 6.1 correctly reports that joint pre-training outperforms interleaved pre-training, and the subsequent SFT comparison uses the same pre-training starting point and underlying input–target pairs. However, the multi-view cycle also introduces additional predicted views and potentially a different training-token and compute budget. It remains unclear how much of the gain comes specifically from coupling predictions in one sequence rather than from additional multi-view supervision. This distinction matters for the central explanation of why the training paradigm works.

2.The main text-to-3D evaluation supports semantic alignment more directly than broad geometric fidelity. Section 5.1 evaluates 100 prompts using ULIP-2 and Uni3D point–text similarities. These are useful measures, but do not by themselves establish improved topology, thin-structure reconstruction, or geometric consistency across a broad range of shapes. The editing experiments provide separate geometric evidence, and the paper references a user study in Supplement C.5; the connection of that study to the text-to-3D claims should be made explicit.

Questions for authors

1.In Table 7, do the single-view and multi-view cycle SFT configurations use the same continued-training checkpoint, optimization schedule, and total training-token or compute budget? Is there a control using the same multi-view supervision without concatenating the predictions into a cycle?

2.Does the user study in Supplement C.5 evaluate text-to-3D geometric quality, instruction-based editing, or both? Please clarify which results support the geometric-fidelity claims in Section 5.1 beyond point–text semantic alignment. 3.Section 3.2 states that 450 source shapes and instructions are reserved for Consis3D-Bench. Were all edits, rendered views, and near-duplicate variants derived from these source shapes excluded from 3DEdit-1M training? Please specify the level at which the split was enforced.

1 comment
AuthorsOpenReview ↗

Rebuttal of Omni123

Thank you for your interest in our work and for the thoughtful and constructive feedback.

1. Controlled comparison of SFT configurations (Q1, W1)

The baseline labeled “Single-view SFT” in Table 7 also uses multi-view images as inputs, with the same viewpoints as the multi-view cycle configuration. Both configurations are trained on the same total number of tokens, matching input-view coverage and the overall training-token budget. We will make these experimental controls explicit in the revised manuscript to clarify the comparison.

2. Geometric quality evaluation and user studies (W2)

Thank you for highlighting the need to clarify geometric-quality evaluations. Following existing 3D foundation models such as TRELLIS and ShapeLLM-Omni, we report evaluation metrics while excluding those that require ground-truth 3D assets. Since Omni123 is the first multimodal model that unifies image and 3D generation, we mainly focus on overall quality and alignment metrics in our exploration. Another reason is that there is still lack of reliable topology and detailed geometry quality metrics in 3D generation. Thus we conduct the user study to further verify our text-to-3D claims by giving user the following instructions:

Instruction for Text-to-3D generation evaluation

Please compare the generated 3D models with the text prompt and select the best overall result, considering geometric quality, surface smoothness, and text alignment together.

  • Geometric quality: Examine shape completeness, proportions, and geometric detail. Look for defects such as fragmented mesh surfaces, floating geometry, self-intersections, or incorrectly fused parts.
  • Surface smoothness: Check for unwanted roughness, irregular bumps, and surface discontinuities, while considering whether intentional sharp edges and fine details are preserved.
  • Text alignment: Assess how faithfully the model represents the objects, attributes, part configurations, and spatial relationships specified in the prompt.

3. Training–test separation (Q2)

The split is enforced at the source-shape level during data synthesis. The 450 source shapes used to construct Consis3D-Bench are totally held out from 3DEdit-1M. All editing pairs, rendered views, and near-duplicate variants derived from these source shapes are excluded from the training set.

Comments on the paper

1 comment

Dear Reviewers, as the discussion deadline is approaching, please review the authors’ responses and join the discussion. Please share whether your concerns have been addressed and update your assessments as appropriate. Thank you for your time and contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.