acceptodds
Withdrawn from ICLR 2027

Hybrid-Diffusion: Latent-to-Pixel Diffusion for Limited Data Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pixel diffusion models have achieved remarkable success in synthesizing fine textures and local structures by denoising score matching. Despite this success, learning accurate score functions in such a high-dimensional space requires substantial data, making it difficult to reliably capture global semantic and structural information under limited data regimes. Latent diffusion addresses this challenge by removing pixel-level redundancy and performing diffusion in a compressed representation that retains salient semantic and structural information. However, because the autoencoder is typically frozen and lossy, this compression can limit the fidelity of fine-grained details. In this paper, we introduce Hybrid Diffusion, a hybrid latent-to-pixel diffusion framework that effectively combines the advantages of both pixel and latent diffusion models for limited data generation. Specifically, Hybrid Diffusion consists of both latent and pixel stages, and performs most of the early denoising in the latent stage before switching to the pixel stage at a chosen transition point, thereby substantially narrowing the pixel-space learning problem while addressing the representation bottleneck of latent space. Importantly, we make the pixel stage explicitly aware of latent prediction errors by using decoded clean predictions from randomly forward-corrupted training latents as training conditions, allowing the pixel stage to address these errors alongside autoencoder reconstruction artifacts. Experiments across multiple datasets demonstrate consistent improvements over both pixel and latent diffusion baselines.

Primary area:
generative models
Keywords:
Diffusion Models, Limited Data

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iod2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hybrid Diffusion, a latent-to-pixel diffusion framework for image generation under limited-data regimes. The method performs most of the early denoising process in the latent space of a frozen VAE, then switches at a predefined noise level to a conditional pixel-space denoiser for low-noise refinement. To reduce the mismatch between training and inference, the pixel stage is conditioned on decoded predictions produced by the latent denoiser rather than on clean VAE reconstructions. Experiments on several low-shot datasets, FFHQ subsets, and BRECAHAD show improvements over multiple pixel-space and latent-space diffusion baselines.

Critical strengths
  • The paper addresses a meaningful problem and presents a simple, practical formulation. The complementary behavior of latent and pixel diffusion is particularly relevant in limited-data regimes: latent modeling reduces the difficulty of learning global structure, while low-noise pixel refinement can recover details constrained by a frozen VAE.
  • The empirical evaluation is broad across limited-data settings. The paper evaluates multiple low-shot datasets, FFHQ subsets, and a higher-resolution medical dataset, and additionally demonstrates that the framework can be applied to a SiT backbone.
Critical weaknesses
  • The methodological novelty is not sufficiently distinguished from prior cascaded and latent-to-pixel diffusion approaches. The general strategy of obtaining a coarse or latent representation before pixel-space refinement has substantial precedent, including DiffuseVAE [2], Latent Forcing [3], Hi-DiT [4], and PiD [5]. Moreover, the proposed model-produced conditioning is closely related in motivation to conditioning augmentation in Cascaded Diffusion Models [1], since both aim to expose the downstream stage to imperfections introduced by the preceding generation stage. The paper does not sufficiently clarify what is fundamentally new beyond adapting these ideas to limited-data generation.
  • The experimental validation does not include the most direct alternatives to the proposed method. The main baselines are predominantly pure pixel-space or pure latent-space diffusion models, so the experiments demonstrate that a hybrid approach can be beneficial but do not establish that this particular hybrid design is preferable to existing latent-to-pixel or generator-refiner approaches. This concern is particularly important because the paper's central contribution is precisely the latent-to-pixel transition and cross-stage conditioning strategy. Broader side-by-side qualitative comparisons would also help verify the claimed recovery of fine-grained details.

References

[1] Ho, J.; Saharia, C.; Chan, W.; Fleet, D. J.; Norouzi, M.; and Salimans, T. 2022. Cascaded Diffusion Models for High Fidelity Image Generation. Journal of Machine Learning Research.

[2] Pandey, K.; Mukherjee, A.; Rai, P.; and Kumar, A. 2022. DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents. Transactions on Machine Learning Research.

[3] Baade, A.; Chan, E. R.; Sargent, K.; Chen, C.; Johnson, J.; Adeli, E.; and Fei-Fei, L. 2026. Latent Forcing: Reordering the Diffusion Trajectory for Pixel-Space Image Generation. ICML.

[4] Shen, Y.; Li, Y.; Pan, Y.; Zhang, Y.; and Yao, T. 2026. Hi-DiT: Hybrid Latent-Pixel Diffusion Transformer for Image Generation. ECCV.

[5] Lu, Y.; Wu, Q.; Wu, J. Z.; Wang, Z.; Ling, H.; Fidler, S.; and Ren, X. 2026. PiD: Fast and High-Resolution Latent Decoding with Pixel Diffusion. NeurIPS.

Questions for authors
  • How does the proposed model-produced conditioning differ conceptually and empirically from conditioning augmentation in Cascaded Diffusion Models [1]? In particular, have the authors compared against a simpler baseline that directly corrupts the clean decoded condition rather than producing the condition through the latent denoiser?
  • Can the authors provide comparisons against more direct latent-to-pixel or generator-refiner approaches, such as DiffuseVAE [2], Latent Forcing [3], Hi-DiT [4], or PiD [5]? Such comparisons would materially affect my assessment of the novelty and effectiveness of the proposed hybrid design.
  • Can the authors provide broader side-by-side qualitative comparisons with latent and hybrid baselines? Since a central claim is that the pixel stage recovers details lost by latent diffusion, direct visual evidence across multiple datasets would help determine whether the improvement comes from the proposed refinement mechanism.
  • What is the additional training cost relative to the latent baseline? The inference-time analysis is useful, but the proposed training pipeline requires additional latent prediction, VAE decoding, and pixel denoising. Reporting training time or GPU-hours and peak memory would help assess the practical trade-off.
Reviewer_RXV8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Hybrid Diffusion for image generation with limited data. It performs most denoising in latent space, then applies conditional pixel-space refinement to the decoded prediction. The pixel model is trained using decoded latent predictions as conditions, with the aim of correcting reconstruction artifacts and latent-prediction errors. Experiments evaluate generation quality across low-shot datasets, FFHQ subsets, and BRECAHAD, with additional component ablations and a SiT backbone evaluation.

Critical strengths
  1. The overall idea is easy-to-follow.
  2. Quantitative experimental comparisons are abundant and the performance of proposed approach is good.
Critical weaknesses
  1. The overall technical contribution is limited, which is mostly a post-refinement in pixel space.
  2. The claimed "error-aware conditional refinement" (line 94-95) lacks direct validation experiement nor theoretical analysis.
  3. Qualitative comparisons with existing approaches are lacking except one sample from figure 1.
  4. It is unsure why pixel-space refinement would help under limited data scenario. The analysis presented in section 3.1 is somewhat confusing.
Questions for authors

See above.

Reviewer_RDthOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hybrid Diffusion, a latent-to-pixel diffusion framework for limited-data image generation that combines latent-space denoising with error-aware conditional pixel-space refinement to improve generation quality.

Critical strengths
  1. Effective performance in limited-data generation. The proposed method achieves notable improvements over both pixel-space and latent-space diffusion baselines across multiple datasets, including low-shot face generation and medical image synthesis. The consistent gains in FID and other evaluation metrics demonstrate the practical effectiveness of combining latent generation with pixel-space refinement.

  2. Reasonable error-aware conditioning strategy. Instead of relying on ideal auto-encoder reconstructions, the method uses decoded predictions from noisy latent inputs to condition the pixel refiner. This allows the model to account for both auto-encoder reconstruction artifacts and latent prediction errors, and the ablation studies support the effectiveness of this design.

Critical weaknesses
  1. Limited novelty compared with existing latent-to-pixel diffusion methods. The proposed framework closely resembles existing latent-to-pixel diffusion and diffusion-based image refinement approaches, where pixel-space diffusion is used to improve the quality of latent-generated images. The main contribution appears to be the error-aware conditioning strategy rather than a fundamentally new framework. However, the paper lacks sufficient comparisons with related approaches, such as PiD, making it difficult to assess the novelty and benefits of the proposed design.

  2. Insufficient evaluation of generalization and computational efficiency. The experiments primarily focus on relatively constrained datasets, such as human and animal faces, without evaluating more diverse and challenging distributions like ImageNet. Moreover, the proposed method introduces an additional pixel diffusion model, but its training and inference costs are not sufficiently compared with stronger latent-only baselines under matched computational budgets. These limitations make it difficult to assess the scalability and practical advantages of the framework.

Questions for authors
  1. Novelty compared with existing latent-to-pixel diffusion methods. What are the key methodological differences between Hybrid Diffusion and existing latent-to-pixel diffusion frameworks, such as PiD? Beyond the error-aware conditioning strategy, what advantages does the proposed approach offer over conventional diffusion-based image refinement or super-resolution methods?

  2. Generalization to more diverse datasets. The experiments primarily focus on human faces, animal faces, and medical images. Can the authors evaluate Hybrid Diffusion on more diverse and challenging datasets, such as ImageNet under limited-data settings, to demonstrate whether the proposed approach generalizes beyond relatively constrained image distributions?

  3. Computational efficiency and practical benefits. Since Hybrid Diffusion requires an additional pixel-space diffusion model, what are the total training and inference costs compared with latent-only diffusion? Can the authors provide comparisons under matched computational budgets to establish whether the improvements justify the additional complexity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.