acceptodds
Withdrawn from ICLR 2027

PixelMoE: Scaling Pixel-Space Diffusion with LLM-Style Mixture-of-Experts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Pixel-space diffusion models provide an appealing alternative to latent-space diffusion models by directly modeling raw images without relying on a learned tokenizer. However, scaling pixel-space models has predominantly followed dense strategies, tightly coupling model capacity with active computation. In this work, we show that standard LLM-style Mixture-of-Experts (MoE) architectures can be effectively transferred to pixel-space diffusion. Building on this finding, we introduce **PixelMoE**, a pixel-space generative framework that sparsely scales model capacity with standard LLM-style MoE, without diffusion-specific routing mechanisms or auxiliary routing objectives. Specifically, PixelMoE combines standard Token-Choice (TC) routing with fine-grained routed experts for input-adaptive sparse computation, a shared expert for capturing common representations, and a bias-based load-balancing strategy. To complement sparse capacity scaling, PixelMoE further introduces Pixel-Space Hierarchical Flow Matching, which varies spatial resolution along the denoising trajectory from global structure formation to fine-grained detail refinement. At comparable active parameters, PixelMoE consistently outperforms its dense counterparts across model scales, demonstrating effective sparse capacity scaling in pixel space. On class-conditional ImageNet-1K, PixelMoE achieves FID scores of 1.49 at 256x256 and 1.56 at 512x512 without representation alignment or perceptual losses, comparing favorably with leading pixel-space and latent-space generative models.

Primary area:
generative models
Keywords:
Pixel-Space Image Generation, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_ibUnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies sparse capacity scaling for pixel-space diffusion. It combines LLM-style token-choice MoE, shared experts, and bias-based load balancing with hierarchical flow matching and stage-wise endpoint prediction. Experiments on ImageNet at 256 and 512 resolutions show improved FID over controlled dense baselines and benefits from increasing the expert pool at a fixed active-parameter budget.

Critical strengths
  1. The controlled sparse-scaling experiments are a strength. The dense baselines share the hierarchical flow setup, prediction head, optimizer, and training recipe. Table 1 shows consistent improvements across three scales, including 2.95 to 1.78 FID at the Large scale with comparable active parameters. Table 2a further shows improvements when the expert pool grows while active parameters remain fixed. These results provide useful evidence that the proposed recipe benefits from sparse capacity.

  2. The method achieves competitive image quality without a learned tokenizer, representation alignment, or perceptual supervision, reporting FID 1.49 at 256 resolution and 1.56 at 512 resolution. This is a useful result for self-contained pixel-space generation.

Critical weaknesses
  1. The incremental contribution over the cited prior work remains limited and insufficiently distinguished. JiTMoE (Liu et al., 2025) already applies LLM-style shared and routed experts with token-choice routing and bias-based balancing to pixel-space diffusion; multi-resolution flow paths and scale transitions are established in Pyramidal Flow Matching and PixelFlow. Although the present configuration produces substantially stronger results, its comparison with JiTMoE also changes patch size, backbone and expert configuration, prediction head, optimizer, and training duration. The controlled dense comparisons establish the value of sparse capacity within this recipe, but do not clearly identify the new design insight responsible for the advance over earlier recipes. The additional hierarchical-flow improvement in Table 9b is small (2.54 to 2.46). I find the methodological and conceptual advance less compelling than the empirical performance.

  2. The practical resource trade-off is insufficiently quantified. The paper reports active parameters and training epochs, but not FLOPs, wall-clock cost, throughput, or memory. In particular, the 2.8x convergence comparison in Figure 5 measures epochs. The MoE model stores substantially more parameters and incurs routing overhead, so these proxies do not establish the end-to-end cost advantage. A representative comparison at matched generation quality would help determine the practical significance of this otherwise incremental design.

Questions for authors
  1. What is the principal new design insight beyond the cited JiTMoE recipe, and which controlled evidence establishes its contribution? In particular, how should we distinguish the effect of the MoE configuration from changes to token granularity, the backbone, and the training recipe?

  2. For a representative scale such as Large, what are the training time and peak memory required to reach a matched FID, and the sampling throughput, relative to the controlled dense baseline? Please specify the hardware and the final CFG, guidance interval, and EMA settings.

Reviewer_hNB3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces PixelMoE, a pixel space diffusion model with LLM-style MoE. The authors directly apply the DeepSeek MoE recipe to diffusion transformers, including token-choice routing, fine-grained experts, a shared expert, and bias-based load balancing. The paper further introduces hierarchical flow matching, where they vary the spatial resolution along the denoising trajectory and splitting denoising into a coarse-to-fine generation. The authors report state-of-the-art results on ImageNet for pixel-space generation, without relying on representation alignment or perceptual losses.

Critical strengths
  • The paper achieves very strong empirical results on class-conditional ImageNet, competitive to current latent space models, and this without any representation alignment or auxiliary objectives.
  • The paper provides a good set of ablations of different MoE design choices.
  • From an empirical standpoint, rendering flow matching hierarchical in a coarse-to-fine manner with MoE also makes sense, which might also give them some inference gain over full resolution denoising
  • The overall approach is relatively straightforward and simple to understand, combining MoE techniques with hierarchical generation.
Critical weaknesses

W1: Limited novelty After reading the paper, I have concerns about its methodological novelty. While it can be a certaintly a methodological contribution to combine previously existing components, the paper does not sufficiently distinguish itself from a relevant prior work. In particular, JiTMoE already uses the same MoE DeepSeek recipe in their paper from December last year, including pixel-space MoE, shared and fine-grained experts and token-choice routing. The hierarchical flow follows Pyramidal Flow Matching (see appendix B). The central message of the paper is also very close to JiTMoE. Albeit there exists a big overlap, the paper only mentions JiTMoE briefly and does not directly separate its contributions from it.

W2: Lack of isolated contributions According to Table 9d, the biggest gain seems to stem from the DDT head (FID 3.76 -> 2.46), which is also confounded, because removing the head removes almost half of the active parameters. In contrast, the gain from e.g. the single-stage to the hierarchical flow (table 9b) is rather modest, but is one of the main claimed contributions of the paper. The remaining differences rather seem like optimization tricks (e.g. gMuon, longer training) and are not ablated in the paper. Hence, it is not clear what actually drives the gains, whether it is the papers claimed contributions or engineering effort. If the improvements mainly come from a more carefully tuned configuration, the contribution is an effective engineering adaptation rather than a novel methodology.

W3: Endpoint prediction conflicts with JiT motivation JiT motivates x-prediction by noting that clean images lie on a low-dimensional manifold, while noisy quantities do not. The paper cites JiT to motivate stage-wise endpoint prediction, but if I understood correctly, the stage 0 endpoint is mostly noise, and therefore violates JiTs central assumption. Only stage 1 seems to predict a clean image.

W4: Training cost is not reported The paper reports no FLOPs, wall-clock time, or GPU-hours. Against its dense counterparts, PixelMoE has more than twice as many total parameters, plus routing and communication overhead. Against the Table 3 baselines, patch size 8 means four times more tokens than patch size 16 models such as JiT. Figure 5 only measures convergence in epochs, but using FLOPs or GPU-hours would make it clearer how much computational gains the method gets.

Minor weaknesses

  • Strong and recent latent-space baselines are missing from Table 3, for example, DDT (whose prediction head is also used in this work), LightningDiT, RAE.
  • HDiT is shown in Figure 1, but neither compared nor discussed.
  • Table 2b indicates that the advantage over dense methods is smaller with larger patch size. JiTMoE whose modest gains motivate this paper, use a patch size of 16. This might be an explanation of the gap to JiTMoE rather than the MoE design choices. A clear separation of its effect is missing.
  • While I do think that ImageNet is a good benchmark to isolate specific contributions, I think scaling it to more difficult and non-saturated tasks like text-to-image is crucial, especially considering that the H model already seemed to require tricks to avoid overfitting.
Questions for authors
  • I would appreciate a more detailed discussion of the differences to JiTMoE. As this method was already published in December last year, a clear empirical justification and ablation of individual components would be great, to isolate which of the papers claims contributed the most to the gains (longer training? smaller patch size?)
  • Why is noisy endpoint prediction preferable to clean-image prediction in Stage 0, given JiTs findings that predicting noisy quantities is harder in pixel space? Is it possible to try a clean-image target instead?
  • What are the training FLOPs, wall-clock time, memory and GPU-hours of PixelMoE compared with its dense counterparts? Can Figure 5 be shown in FLOPs or GPU-hours?
  • How large is the MoE vs dense gap at patch size 16 under the full recipe? Does patch size explain the gap to JiTMoE better than the MoE design does?
Reviewer_tuPiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the integration of the Mixture-of-Experts (MoE) architecture into pixel-space diffusion models to improve the scalability and computational efficiency of image generation models. Traditional pixel-space diffusion models generate images directly in RGB pixel space, avoiding the information loss introduced by VAE compression. However, they face substantial computational costs and challenges in model scaling. The core idea of PixelMoE is to leverage a sparse expert mechanism by replacing the feed-forward networks in Transformers with multiple expert networks and dynamically routing each image token to a small subset of experts. This enables an increase in total model parameter capacity without a proportional increase in computational cost. Such a design has the potential to balance pixel-level image generation quality with efficient model scaling, offering a promising direction for developing larger-scale native pixel-space generative models.

Critical strengths
  1. Simple and scalable architectural design. The paper successfully introduces LLM-style MoE into pixel-space diffusion models, employing standard Token-Choice Routing, fine-grained experts, and shared experts. It does not require diffusion-specific routing mechanisms or additional routing losses, providing an effective sparse scaling strategy for pixel-space generative models.
  2. Effective decoupling of model capacity and computational cost. Through fine-grained expert partitioning and dynamic Token-Choice Routing, the model can substantially increase its total parameter capacity while maintaining a relatively small number of activated parameters. This improves parameter scaling efficiency and provides a feasible approach to exploring larger-scale generative models.
  3. Relatively comprehensive and convincing experimental evaluation. The paper not only demonstrates image generation performance across different resolutions but also investigates the sources of performance improvements through comparisons between dense and MoE models, as well as ablation studies on the number of experts, routing strategies, shared experts, and load balancing. These experiments provide empirical support for the architectural design choices. Nevertheless, greater care is needed to ensure fairness in certain comparisons.
Critical weaknesses
  1. Unclear practical computational efficiency. Although the paper demonstrates advantages in parameter scaling and training convergence, it lacks end-to-end comparisons of training time and inference latency. Consequently, it remains difficult to assess the actual computational speedup achieved by the proposed method.
  2. Premature assessment of training convergence. The paper only reports results obtained after 320 training epochs. Providing additional results from longer training schedules, such as 600 or 800 epochs, would help better characterize the performance ceiling of the proposed method and determine whether its advantages persist with extended training.
  3. Potential bias introduced by the evaluation protocol. Previous studies have pointed out that the evaluation script used by JiT may introduce a systematic discrepancy of up to 0.08 compared with the more commonly adopted ADM evaluation script [1, 2]. Could this discrepancy affect the validity of the experimental results and the conclusions regarding the effectiveness of the proposed method?
  4. Potential confounding effects of different patch sizes. Most existing methods adopt a patch size of 16, resulting in 256 tokens for 256 × 256 images. In contrast, the proposed method uses a patch size of 8. Could this difference lead to a substantial increase in actual computational cost, thereby affecting the fairness of the comparisons? Furthermore, how does the proposed method perform when using a patch size of 16?

[1]Chen H, Ackermann J, Kim M, et al. Asymmetric Flow Models[J]. arXiv preprint arXiv:2605.12964, 2026.

[2]Tong H, He Y, Li F, et al. Energy-Guided Flow Matching[J]. arXiv preprint arXiv:2608.05811, 2026.

Questions for authors

In addition to the issues raised in the Weaknesses section, I have the following questions:

  1. Why are auxiliary alignment techniques such as REPA not incorporated into the model? Have the authors explored similar approaches? According to studies such as PixelREPA [1], incorporating REPA into JiT yields relatively limited improvements, whereas REPA has a more substantial impact on the performance of PixelDiT-style models. It would be valuable to understand whether similar observations hold for the proposed architecture.
  2. Why do the authors adopt Pixel-Space Hierarchical Flow Matching instead of a more unified flow-matching formulation? The proposed approach adjusts the training objectives across different stages. What motivates this design choice over methods that employ a more unified path without requiring repeated noise injection, such as Energy-Guided Flow Matching [2]?

[1]Shin J, Kim J, Shim H. Representation alignment for just image transformers is not easier than you think[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2026: 603-620.

[2]Tong H, He Y, Li F, et al. Energy-Guided Flow Matching[J]. arXiv preprint arXiv:2608.05811, 2026.

Reviewer_NxwQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PixelMoE, a LLM-style mixture of experts approach for pixel diffusion. The approach successfully implements token-based mixture of experts routing for pixel diffusion, without explicit timestep-based or semantic routing. The authors additionally introduce a hierarchical sampling method for coarse-to-fine generation orthogonal to the pixel mixture of experts contributions that additionally improves performance. The authors claim superior results with equivalent active parameters to prior work, and conduct extensive ablations to justify their design decisions.

Critical strengths

This paper's critical strength is a standard method for implementing mixture of experts training on pixel diffusion. The methods introduced are not novel, however correctly implementing mixture of experts for diffusion models has been challenging, and if successful in accomplishing a simple form of mixture of experts training similar to language modeling, this paper will have good impact and usefulness to the community.

Critical weaknesses

The critical weakness of this paper is soundness, in particular whether the gains in this work actually result from PixelMoE training or one of the many other decisions the authors introduce.

After carefully reading the paper, I see several places of concern.

  1. The hierarchical sampling is complicated (1.5 pages), orthogonal to the story of the paper, and provides little to no gain 2.54 -> 2.46 in ablation settings.
  2. The authors halve the patch size (at which hierarchical stage is the patch size reported?) which in general is a 4x increase in FLOPs at training and inference. However, the authors compare using active parameters in their main table (Table 3). This is an unacceptable edit to keep hidden and the authors must report FLOPs and wall-clock inference time for at least one baseline (say, JiT).
  3. The most impactful gain in this paper is from implementing a DDT head for pixel generation, resulting in a 3.76 to 2.46 reduction in FID. To my knowledge, the baselines do not use a DDT head. Therefore it is unclear if gains versus baselines come from MoE or merely more parameters in FLOPs.
  4. There is no direct baseline in the paper versus prior work and only one result at a standard scale for MoE training. The only direct baseline showing the improvement of MoE is in Table 2.b, showing a decrease from 3.15 to 2.89 FID at ablation scale at patch size 16, in a baseline configuration that the authors have decided (With DDT, Hierarchical sampling, 100 epochs). Considering the motivation of the paper for being the first paper to make MoE work for generation, this baseline should compare versus prior work rather than at ablation scale. Would PixelMoE work without the other changes?
  5. Some ablations have confusing results: Why would adding explicit routing guidance harm FID heavily in table 9.a? The tokens already are conditioned on timestep routing. It would make sense that timestep-only routing harmed performance, however adding information should not in general cause such a harm to generation quality, especially when the information is already available in the latent.
Questions for authors

Are the gains in this paper actually from PixelMoE or from one of the many other changes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.