acceptodds
Withdrawn from ICLR 2027

Structured 3D Latents Are Surprisingly Powerful: Unleashing Generalizable Style with 2D Diffusion

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D asset generation plays a pivotal role in fields such as gaming and virtual reality, enabling the rapid synthesis of high-fidelity 3D objects from a single or multiple images. Building on this capability, enabling style-controllable generation naturally emerges as an important and desirable direction. However, existing approaches typically rely on style images that lie within or are similar to the training distribution of 3D generation models. When presented with out-of-distribution (OOD) styles, their performance degrades significantly or even fails. To address this limitation, we introduce DiLAST: 2D Diffusion-based Latent Awakening for 3D Style Transfer. Specifically, we leverage a pretrained 2D diffusion model as a teacher to provide rich and generalizable style priors. By aligning rendered views with the target style under diffusion-based guidance, our method optimizes the structured 3D latent representations for stylization. We observe that this limitation stems not from insufficient model capacity, but from the underutilization of structured 3D latents, which are inherently expressive. Despite being trained on comparatively limited data, 3D generation models can leverage 2D diffusion guidance to steer denoising toward specific directions in latent space, thereby producing diverse, OOD styles. Extensive experiments across diverse data and multiple 3D generation backbones demonstrate the effectiveness and plug-and-play nature of our approach.

Primary area:
generative models
Keywords:
3D Generation, Style Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hEVSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DiLAST for reference-image-guided 3D style transfer. It uses a pretrained 2D diffusion model to guide sampling in the latent space of a pretrained 3D generator. Attention distillation transfers style, while a content loss and two regularization terms preserve structure and reduce artifacts. Experiments show stylization results on TRELLIS and extensions to UniLat3D and Hunyuan3D.

Critical strengths
  • Applying 2D style guidance to a pretrained 3D latent space is useful. The presented results transfer a range of styles while preserving recognizable object structure. The examples on different 3D generators also suggest that the approach is not restricted to TRELLIS.

  • The floater removal and color speckle suppression terms address practical problems during optimization. Figure 5 shows their effects on the generated assets.

Critical weaknesses
  • The main concern is the methodological novelty. Using 2D diffusion priors to optimize 3D representations has been studied extensively, starting with DreamFusion. More closely, DiffStyle3D already uses diffusion attention features to guide 3D style transfer. The style and content losses in Eqs. (7)-(8) are adopted from Attention Distillation, which the paper cites. The main difference appears to be applying this guidance during sampling in a pretrained structured 3D latent space, rather than directly optimizing an existing 3D representation. This is a reasonable adaptation, but the paper should explain more clearly what additional technical contribution it provides.

  • The comparisons do not establish the advantage over existing 2D-diffusion-guided 3D stylization. Table 1 compares with MorphAny3D and StyleSculptor, while Appendix C.2 considers transfer-then-generate approaches. These are useful comparisons, but they do not test whether directly optimizing the generated 3D asset with 2D diffusion guidance would achieve similar results. DiffStyle3D optimizes the color parameters of an existing 3DGS scene, so its original task differs from DiLAST. An adapted comparison starting from the same content-conditioned 3D asset would help show the benefit of optimizing the 3D latent during sampling. The comparison should account for optimization time, given DiLAST's 500-step procedure.

Questions for authors
  • How does DiLAST compare with direct 3DGS optimization, or optimization of the final clean 3D latent, using comparable style supervision and optimization time?
  • How many content-style pairs are evaluated in Table 1, and how are the claimed OOD styles defined?
Reviewer_H85LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DiLAST, a training-free framework for image-guided 3D style transfer. It uses attention distillation from a frozen 2D diffusion model to guide structured 3D latents during rectified-flow sampling. Multi-view differentiable rendering connects the 2D objectives to the 3D latent representation. Additional opacity-consistency and latent-smoothness losses mitigate floating Gaussians and color artifacts. Experiments demonstrate improved stylization over MorphAny3D and StyleSculptor across various artistic references.

Critical strengths
  1. The paper addresses a relevant limitation of existing 3D stylization methods: their difficulty handling artistic references that differ substantially from typical 3D object renderings.
  2. The qualitative results show compelling stylization and better preservation of overall object structure than the evaluated baselines.
  3. The framework works with frozen pretrained models, and the qualitative ablations illustrate the usefulness of the proposed regularization terms.
Critical weaknesses
  1. Missing discussion and comparison with a closely related method. GuideFlow3D (NeurIPS 2025) is neither cited nor included in the comparisons. It already performs training-free 3D appearance transfer by fixing sparse voxel positions, optimizing structured latent features, and interleaving rectified-flow sampling with gradient-based updates. These are central design choices shared by DiLAST. The main distinction is the use of render-based 2D attention distillation instead of GuideFlow3D’s part-aware latent guidance, together with additional regularization. This distinction may be useful, but the current presentation does not adequately establish the contribution beyond this prior framework. A direct comparison with GuideFlow3D is necessary to assess the benefits of the proposed guidance.

  2. Preserving the overall shape of a character does not necessarily preserve its identity. The query-based content loss and opacity constraint do not explicitly address this distinction, and the reported evaluations do not isolate identity preservation. Portrait-focused experiments would help assess whether stylization alters identifying facial details. One useful experiment would first stylize the input portrait in 2D and then use that stylized portrait as the style reference in DiLAST, while retaining the original content and geometry conditions. This would test whether a reference depicting the same person improves identity preservation without weakening the target style.

Questions for authors

The reviewer leans toward Borderline Reject. The visual results are promising, and using 2D diffusion features to guide 3D stylization is practically useful. However, the omission of GuideFlow3D leaves the methodological novelty and empirical advantage over closely related work insufficiently established. Fine-grained identity preservation also requires more targeted evaluation. A convincing comparison with GuideFlow3D and clearer evidence for the benefits of the proposed optimization design could change my assessment.

Reviewer_b8LHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiLAST guides the sampling process of structured 3D latent variables within a pre-trained 3D generative model by leveraging a frozen 2D diffusion model. Implemented primarily atop TRELLIS, this approach preserves the sparse structure generated during the first stage. In the second stage, the 3D latent is decoded into 3D Gaussian Splatting (3DGS) representations at each step, from which multiple viewpoints are rendered. Gradients are then derived via attention distillation and content constraints from the 2D model, which are subsequently used to refine the 3D latent. Furthermore, the authors incorporate silhouette constraints and smoothing of latents across adjacent voxels to mitigate the occurrence of floating artifacts and color distortions.

Critical strengths

(1) It introduces a control mechanism that differs from direct 3D attention injection. Signals from 2D attention distillation are backpropagated—via a frozen 3DGS decoder and differentiable rendering—into the structured 3D latent space. This entire process eliminates the need to retrain either the TRELLIS model or the 2D diffusion teacher. The decision to intervene at specific points along the 3D generation trajectory is the most noteworthy design aspect of this work.

(2) The results demonstrate a clear performance improvement. As shown in Figure 4, DiLAST generally preserves objects more completely than the two baselines; this is further supported by the human and automated preference metrics presented in Table 1, which align with the visual observations. The ablation study on auxiliary losses (Figure 5) and the examples involving real-world objects (provided in the appendix) indicate that the authors have explicitly addressed the issue of prominent artifacts.

Critical weaknesses

(1) The core claims regarding OOD (out-of-distribution) performance and the "underutilized potential of latent variables" lack empirical validation. The paper does not categorize in-distribution versus out-of-distribution styles—relative to 3D backbones like TRELLIS—nor does it investigate the relationship between the degree of distribution shift and model performance. Furthermore, the attention distillation loss is adopted from existing literature; the paper fails to compare the proposed "step-wise guidance" approach against a baseline that optimizes the final, "clean" latent variable using the same teacher model, the same loss function, and a comparable computational budget. While the results demonstrate the overall effectiveness of the proposed method, they do not clarify whether the performance gains stem specifically from interventions during the denoising trajectory or from the 2D teacher providing superior style supervision.

(2) The choice of baselines and evaluation protocols makes it difficult to assess the true performance gap. Many of the baseline results in Figure 4 are significantly incomplete—which may reflect genuine limitations of those methods under the given conditions—yet the paper lacks essential details such as checkpoint information, hyperparameter tuning procedures, or failure rates. The human evaluation is based on only ten participants, which is arguably insufficient. Additionally, given that the method requires 500 sampling steps, repeated multi-view rendering, and backpropagation through a 2D teacher model, the paper fails to report end-to-end execution time or GPU memory consumption.

Questions for authors

Q1. How do the authors define and verify "out-of-distribution" (OOD) styles? More importantly, given the same 2D teacher model, loss function, and computational budget, how do the results of directly optimizing the final "clean" latent compare to those obtained via guided iterative sampling? This comparison is the most likely factor to shift my perspective on the contributions of this method.

Q2. Could you provide full evaluation details—including the number of style combinations, baseline configurations and failure rates, the randomization order for human/VLM evaluations, and confidence intervals? Furthermore, please provide separate assessments for content preservation, reference style alignment, and the quality of perspectives not included in the optimization process.

Q3. Regarding the intermediate latent in Equation (9), is it decoded directly, or is the "clean" state estimated first? Are the gradients backpropagated through the Flow Transformer? Additionally, please report the total runtime and VRAM usage, and clarify any implementation differences across the various 3D backbones used.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.