acceptodds
Under review as a conference paper at ICLR 2027

Advanced Pixel Diffusion Model with Guided Sparse Global Refinement

Abstract

Pixel-space diffusion has recently emerged as a promising direction for high-fidelity image generation by modeling images directly in the original pixel domain. However, pixel-space diffusion is computationally demanding due to the extremely high dimensionality of natural images. For efficiency, existing pixel diffusion models either compromise fine details with large-patch tokenization or confine subsequent refinement within individual patches. Such intra-patch refinement inevitably restricts structural continuity across patch boundaries and long-range token interactions, limiting refinement quality. To address these issues, we propose PixSGR, a novel Pixel diffusion framework with Sparse Global Refinement tailored for modeling the distribution of natural images directly in pixel space. PixSGR starts from a supervised low-channel bottleneck to efficiently capture the low-dimensional manifold of natural images. It then progressively expands the channel dimensionality and spatial resolution to recover increasingly fine-grained structures. At the spatial refinement stage, coarse-scale attention maps preselect globally relevant interactions to pre-sparsify fine-scale attention, enabling non-local refinement beyond isolated patches without the quadratic cost of dense attention. Extensive experiments on ImageNet validate the effectiveness of PixSGR. It achieves an FID of 1.51 at 256256 and maintains performance when scaled to 512512, attaining an FID of 1.60.

Primary area:
generative models
Keywords:
pixel diffusion, diffusion model, image generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TUX6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the pixel spatial diffusion framework PixSGR, which aims to balance global structure modeling, detail recovery, and computational cost. The model first learns a compact representation through a low-channel bottleneck with auxiliary supervision, then expands the channels, and utilizes the attention map from the previous coarse-scale stage to select sparse global interaction objects for the fine-scale stage.

Critical strengths
  1. By using coarse-scale attention to guide fine-scale computation, cross-patch interactions can be introduced while controlling computational load, providing a reasonable solution for detail recovery in pixel diffusion.

  2. Results at two resolutions support the effectiveness of the framework and also demonstrate that the method has a certain resolution scaling capability.

  3. The ablation experiments are comprehensive. The authors analyzed bottleneck supervision, attention forms, top-k budgets, mapping strategies, and training objectives.

Critical weaknesses
  1. The paper cites SiD2 but does not include it in the results table. Publicly available results for SiD2 achieve FID values ​​of 1.38 and 1.48 on ImageNet 256×256 and 512×512, respectively, which are lower than the results presented in this paper.

  2. In Table 2(e), with the improved architecture fixed and IG disabled, changing the training objective reduced FID from 68.4 to 33.9, indicating that the training formulation had a significant impact. Currently, there is a lack of JiT controls using the same pseudo-Huber, LPIPS, REPA, and sampling settings, making it difficult to fully determine how much of the final advantage comes from the new architecture.

  3. The paper primarily reports GFLOPs, parameter count, and training epochs, without providing end-to-end latency, throughput, GPU memory usage, or training time. The complete architecture in Table 2(e) has 51.44 GFLOPs, slightly higher than JiT's 50.51 GFLOPs, directly supporting quality improvements at similar computing power. The overhead of dynamic selection, indexing, and data transfer needs to be confirmed through actual measurements.

Questions for authors
  1. Compared to QuadTree Attention and BiFormer, what are the key technical differences in reusing coarse-scale attention across layers? Could you add a comparison with an existing dynamic routing method using the same backbone and budget?

  2. Given the same hardware, precision, batch size, and number of sampling steps, what are the actual latency, throughput, and peak VRAM usage? Do the timing measurements include top-k operations, index generation, data movement, and IG (index-guided) processes?

Reviewer_LKcGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission focuses on improving the generation quality and efficiency of pixel-space diffusion models. The submission points out that using large-patch tokenization to reduce computational costs may lead to the loss of fine-grained details. To mitigate this, the work proposes PixSGR, a coarse-to-fine pixel diffusion framework: 1) PixSGR first introduces a supervised low-channel bottleneck to learn compact image representations and 1) then progressively expands the channel and spatial dimensions for finer image generation. The core component is a guided sparse global refinement mechanism, which uses attention maps from coarse-scale features to select relevant tokens for fine-scale attention. This allows the model to capture long-range dependencies without performing expensive dense attention. Experiments on ImageNet show that PixSGR achieves an FID of 1.51 at 256×256 resolution and 1.60 at 512×512 resolution.

Critical strengths
  • Clear motivation and problem formulation. This submission identifies a relevant limitation of existing pixel-space diffusion models: while large-patch tokenization reduces computational costs, refinement within individual patches may limit cross-patch interactions and affect image details. The proposed method for alleviating this issue is reasonable to me.

  • Coarse-to-fine attention design is good. The proposed sparse attention mechanism uses coarse-scale attention maps to guide token selection at higher resolutions. This is a simple way to introduce global interactions without the full cost of dense attention. The paper also considers different cross-scale mapping strategies and examines their effects on generation quality.

  • Experimental results are promising. The method achieves an FID of 1.51 on ImageNet 256×256, outperforming several recent pixel-space diffusion models (PixelDiT, PixelU, and AsymFlow). Also, the submission provides ablations on bottleneck supervision, sparse attention, token selection, and convolutional upsampling. The comparisons with window and neighborhood attention are useful for understanding the benefit of the proposed global refinement mechanism.

Critical weaknesses
  • Please clarify the key difference from existing works. The main contribution appears to be using coarse-scale attention to guide fine-scale token selection. However, the paper does not sufficiently explain how this differs from existing attention-guided sparse computation or hierarchical attention methods.

After rebuttal: the discussion on BiFormer and QuadTree-A is helpful. This concern is well handled.

  • Insufficient analysis of the coarse-to-fine attention guidance. The assumption is that attention patterns at coarse resolutions can identify the most relevant tokens for fine-scale refinement. But coarse-scale attention may overlook small objects or fine details that only become important at higher resolutions. Please provide a further analysis that would help justify this key design choice.

After rebuttal: the rebuttal is fine. I have reservations for this concern: coarse attention scores to guide fine-scale candidate selection, without dense fine-scale scoring. Some fine details can be missed.

  • Add more evidence for the claimed computational efficiency. The main comparisons focus primarily on generation quality and model size. Although GFLOPs are reported in the ablation studies, the paper lacks a detailed comparison of inference latency, memory consumption, and computational costs against recent pixel diffusion models.

After rebuttal: Forward latency is helpful. Please include in the revision. This concern is addressed.

Questions for authors

How reliable are coarse-scale attention maps for selecting important fine-scale tokens? For example, could the authors analyze whether the proposed selection mechanism misses small objects or fine details that are not strongly represented at coarse resolutions

Reviewer_ZaovOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PixSGR, a pixel-space diffusion model built on JiT (x-prediction with large-patch tokenization). The model has three stages:

  1. Low-channel bottleneck. The first few Transformer blocks run at 1/4 width. An auxiliary head on the bottleneck output predicts the clean image, supervised by Lbot\mathcal{L}_{bot}. At sampling time, this bottleneck prediction also serves as a weak model for "internal guidance" (IG), extrapolating the final prediction away from it at no extra NFE cost.
  2. Channel expansion followed by standard JiT blocks.
  3. Spatial refinement at 2× token resolution with sparse attention. For each fine token, top-k key indices are pre-selected from the coarse-scale attention map of the preceding stage. They are mapped to fine tokens through coarse-to-all, coarse-to-one, or hybrid mappings, so refinement is not confined to individual patches. A convolutional upsampling head replaces unpatchify.

Training combines a pseudo-Huber flow-matching loss with LPIPS and REPA. PixSGR-H/16 reaches FID 1.51 on ImageNet 256×256 (320 epochs, 875M parameters) and 1.60 on ImageNet 512×512. Ablations at B scale (50 epochs) cover bottleneck supervision, sparse vs. window/neighborhood attention, top-k budgets, mapping strategies, and training objectives. An appendix analysis reports that coarse-guided selection retains 48.7% of the fine-scale attention mass, compared with 70.9% for an oracle.

Critical strengths
  1. Strong empirical results for pixel-space diffusion. PixSGR-H/16 reports FID 1.51 on ImageNet 256×256 after 320 epochs with 875M parameters. This beats recent pixel-space baselines such as AsymFlow (1.57), FrequencyBooster (1.60), and PixelDiT (1.61) with fewer epochs or parameters, and it remains competitive at 512×512 (FID 1.60). If the gains hold under matched comparisons, they would be a useful data point for the pixel-diffusion community, so this is the main factor in the paper's favor.

  2. A simple, practical design for cross-patch refinement. Reusing the attention map already computed at the coarse stage to select sparse fine-scale keys adds no extra scoring cost and directly targets a real limitation of recent two-stage pixel diffusion models, namely intra-patch-only refinement. Internal guidance from the bottleneck head is likewise free in NFE. The appendix analysis of attention-mass coverage (Table 6) is a good step toward validating the selection mechanism.

Critical weaknesses
  1. The gain from the proposed sparse global refinement is not isolated from the training recipe. Table 2(e) shows that most of the improvement comes from components adopted from prior work, mainly LPIPS (64.7→37.3), together with REPA and internal guidance. Adding sparse attention gives 74.7→70.3, but this step introduces an entire refinement stage at 2× token resolution (+5.5 GFLOPs) compared with no refinement at all. It therefore shows that a refinement stage helps, not that the coarse-guided selection itself does. The only direct comparison, against window/neighborhood attention (Table 2(b)), shows about 1 FID. Three baselines are missing: dense fine-scale attention, random sparse attention with the same budget, and JiT-H trained with the same objective and guidance. The efficiency motivation would also benefit from measurements: with 1024 fine tokens and 2 blocks, it is unclear whether sparse attention is faster than dense attention in practice, and no wall-clock or memory numbers are reported. Without these comparisons, it is hard to attribute the state-of-the-art result to the paper's main contribution.

  2. Ablations are limited in scale and statistical support. All ablations use the B-scale model trained for 50 epochs with batch size 128, where FID is in the 10–80 range, so it is unclear whether the conclusions transfer to the H-scale, 320-epoch setting. Many differences are small, e.g., 10.1–10.7 across top-k budgets and mapping strategies (Table 2(c)(d)), and no results over multiple seeds are reported. The evaluation protocol also differs across sub-tables: (a)–(d) use CFG, (e) uses 10K samples without CFG, and the sample count for (a)–(d) is not stated. As a result, it is difficult to judge which design choices actually matter.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.