Semantic Tokens as the Context of Pixel Space Generation

Context is central to how large language models generate: what has been produced earlier conditions everything produced later, and this accumulated context is what allows them to solve difficult tasks reliably. Pixel-space diffusion models, in contrast, carry almost no explicit context. The only state passed from one denoising step to the next is the noisy image itself, so any higher-level understanding of what is being generated must be re-derived from pixels at every step. This raises a natural question of what should serve as context for pixel generation. We find that coarse, locally aggregated semantic tokens make an effective context for pixel generation. Unlike a text prompt, this image-specific context is not available at the start of generation. We therefore propose **SemPix**, which jointly generates semantic tokens and pixels from noise through flow matching and carries the semantic state across denoising steps, allowing the evolving context to inform later pixel predictions. Pixel and semantic tokens interact through joint self-attention in a shared Transformer backbone, with separate heads mapping them to their respective targets. Experiments show that this generated context substantially improves pixel generation, and that both its form and its evolution matter. Locally aggregated features are more effective than dense or global alternatives, while the context is most useful when it persists across denoising steps and directly informs pixel prediction. On class-conditional ImageNet-256, SemPix-XL reaches 4.93 unguided FID after 40 epochs and 1.72 guided FID after 160 epochs. These results suggest that, as in language modeling, what a pixel generator carries along its trajectory matters as much as how it predicts the image.