acceptodds
Under review as a conference paper at ICLR 2027

Semantic Tokens as the Context of Pixel Space Generation

75 views

Abstract

Context is central to how large language models generate: what has been produced earlier conditions everything produced later, and this accumulated context is what allows them to solve difficult tasks reliably. Pixel-space diffusion models, in contrast, carry almost no explicit context. The only state passed from one denoising step to the next is the noisy image itself, so any higher-level understanding of what is being generated must be re-derived from pixels at every step. This raises a natural question of what should serve as context for pixel generation. We find that coarse, locally aggregated semantic tokens make an effective context for pixel generation. Unlike a text prompt, this image-specific context is not available at the start of generation. We therefore propose **SemPix**, which jointly generates semantic tokens and pixels from noise through flow matching and carries the semantic state across denoising steps, allowing the evolving context to inform later pixel predictions. Pixel and semantic tokens interact through joint self-attention in a shared Transformer backbone, with separate heads mapping them to their respective targets. Experiments show that this generated context substantially improves pixel generation, and that both its form and its evolution matter. Locally aggregated features are more effective than dense or global alternatives, while the context is most useful when it persists across denoising steps and directly informs pixel prediction. On class-conditional ImageNet-256, SemPix-XL reaches 4.93 unguided FID after 40 epochs and 1.72 guided FID after 160 epochs. These results suggest that, as in language modeling, what a pixel generator carries along its trajectory matters as much as how it predicts the image.

Primary area:
generative models
Keywords:
Pixel space generation, Flow matching, Image generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7VnHOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SemPix, a pixel-space image generation model that jointly generates pixels and a compact set of semantic tokens. The semantic tokens are carried across denoising steps and provide context for subsequent pixel predictions. Their training targets are obtained by applying parameter-free average pooling to frozen DINOv2 features. Pooling a 16×16 feature grid over 2×4 neighborhoods produces 32 continuous tokens that retain spatial structure. The pixel and semantic tokens are processed by a shared Transformer, with separate refinement heads predicting the two targets. No visual encoder is needed at inference. On class-conditional ImageNet-256, SemPix-XL/2 achieves an FID of 1.72 and an IS of 316.1 with classifier-free guidance after 160 epochs. At a matched 40-epoch budget, SemPix-L/2 improves FID from 7.53 to 5.33 relative to PixelGen-L/2, with essentially unchanged reported per-step time and memory usage. The ablations examine both the choice of semantic representation and how it is used during sampling. Additional results on ImageNet-512 and text-to-image generation are provided in the appendix.

Critical strengths

1.The ablations are carefully controlled. The matched-length comparisons help separate the benefit of coarse semantic representations from that of using fewer tokens. The interventions in Table 4 also test whether the semantic state needs to persist across steps and whether pixel tokens need access to it. These experiments provide useful evidence for the proposed mechanism.

2.The finding that coarse, spatially organized semantic tokens work better than either dense features or a single global token is interesting. The paper tests this choice with different encoders and integration mechanisms, which makes the observation more convincing than a result from one configuration.

3.The improvements come with little additional per-step overhead in the reported comparison. The reduction from 7.53 to 5.33 FID at the same training budget is substantial. The XL model also reaches competitive results with 687M parameters, compared with 918M for V-Co-XL/2. The absence of an external visual encoder at inference is another useful practical property.

4.The paper is generally clear and easy to follow. The architecture and experimental motivation are explained well, and the positioning table in Appendix E helps distinguish the method from REG, ReDi, and V-Co.

Critical weaknesses

1.My main concern is whether the paper establishes its conceptual novelty clearly enough. Representation Forcing appears closely related at a high level: its abstract describes using predicted visual representations as context for pixel-space generation without a VAE bottleneck. However, the paper does not cite or discuss it. The comparison with REG, ReDi, V-Co, and Latent Forcing is helpful, but I would still like the authors to explain how SemPix differs from Representation Forcing in its formulation, architecture, and use of the semantic state. Without that discussion, it is difficult to judge how much of the contribution comes from a new generation mechanism and how much comes from the particular representation and implementation choices.

2.The construction of the 32 semantic tokens is simple average pooling, so the novelty does not appear to lie in the compression operation itself. The related-work discussion would benefit from a clearer connection to semantic tokenizers and token-reduction methods. For example, SEED also compresses a 16×16 feature grid into 32 tokens for generation, although its learned, causal, quantized representation differs substantially from SemPix’s continuous spatial tokens. Work such as BEiT v2/VQ-KD is also relevant to the use of teacher features as semantic targets. These connections do not negate the paper’s empirical findings, but discussing them would help identify the specific contribution more precisely.

3.The experiments support the method within the settings studied, but the broader claims about semantic context are less well established. Most of the evidence comes from class-conditional ImageNet-256 with a fixed 50-step sampler. There is no evaluation with fewer sampling steps or after distillation. The COCO text-to-image experiment is useful, but it only compares against a pixel-only baseline. I would prefer the conclusions to distinguish the findings demonstrated by these experiments from the broader motivation for the method.

4.There are a few minor presentation issues. The limitations are spread across the reproducibility statement and appendix discussion, which makes them harder to locate. The ImageNet-512 and text-to-image results are also confined to the appendix. A brief discussion of these results and their limitations in the main text would give a more complete picture of the method’s scope.

Questions for authors

1.How does SemPix differ from Representation Forcing in the way visual representations are predicted and used to guide pixel generation? Please discuss the relationship and, if feasible, include an empirical comparison.

2.How should the proposed tokens be understood relative to SEED-style semantic tokenizers and existing token-reduction methods? Is parameter-free average pooling important to the observed gains, or might a learned compressor or quantized representation work equally well?

3.Why was an asymmetric 2×4 pooling neighborhood chosen? How sensitive are the results to the neighborhood shape and token count? In particular, Table 6 suggests that 64 tokens perform slightly better than 32.

4.Do the gains persist with fewer sampling steps or after classifier-free guidance distillation? A comparison of FID against sampling steps for PixelGen and SemPix would help answer this.

5.Can the text-to-image experiment include stronger pixel-space baselines? This would make it easier to assess the significance of the COCO result.

11 comments

We thank the reviewer for recognizing our controlled ablations, the effectiveness of coarse spatial semantic context, and the low measured computational overhead. We address the questions below, including new experiments with fewer sampling steps and stronger MS-COCO text-to-image baselines.

Q1. Relationship to Representation Forcing (RF).

Thank you for highlighting this related work. Both methods use generated visual representations for pixel generation. RF autoregressively predicts discrete visual representation tokens before pixel diffusion and uses the completed representations as fixed context. SemPix jointly generates continuous semantic states and RGB pixels from noise through flow matching, enabling bidirectional refinement throughout generation.

The architectures also differ. RF builds on a pretrained Qwen3 Mixture-of-Experts language model with a Mixture-of-Transformers architecture, a jointly trained DINOv3 understanding encoder, and online vector quantization. SemPix uses a shared generative Transformer with dual refinement heads and semantic targets from frozen DINOv2 (Sec. 3, Table 8).

These designs serve different research goals: RF targets unified multimodal understanding and generation through an autoregressive framework, whereas SemPix investigates how evolving semantic representations serve as context for direct pixel-space generation. Our work establishes two key findings that RF does not systematically investigate:

  1. Novel findings on effective semantic context. RF uses 2×2 spatial pooling without systematically studying semantic granularity. We demonstrate that coarse spatial semantics outperform global and dense representations as generated context (Table 6).

    This advantage holds at matched sequence length, across integration mechanisms (Table 7), visual encoders (Table 13), backbone architectures (Table 11), and longer training schedules (Fig. 7). Coarse semantic representations are also easier to recover under high noise (Semantic predictability across denoising timesteps (Fig. 6)).

  2. New insights into context evolution and utilization. We show that semantic context is most effective when carried across denoising steps and directly used for pixel prediction. Table 4 validates the importance of both cross-step persistence and semantic-to-pixel interaction.

    Pixel–semantic consistency (Fig. 5) shows that the evolving state tracks the particular generated image beyond class conditioning. Our quantitative Evolution of image and semantic-token predictions during free sampling tracks image and semantic-token predictions, showing earlier agreement with each model's eventual generated image and progressive refinement of SemPix's semantic state.

We will clarify these distinctions and discuss RF in the revised manuscript.

Q2. Semantic token construction and relation to existing tokenizers.

Thank you for suggesting this tokenizer comparison. Our contribution is the generation and use of continuously evolving semantic context in pixel-space generation, rather than the average-pooling operator itself. We systematically study the effective form of this context and its role throughout the generation trajectory (Tables 4, 6–7).

SEED learns 32 quantized, causally ordered visual tokens for autoregressive multimodal generation; SemPix instead maintains continuous, spatially organized semantic states that are jointly generated and refined with pixels. BEiT v2/VQ-KD discretizes teacher features as targets for masked image modeling, rather than carrying them as evolving generative states.

We use parameter-free local pooling to preserve regional semantic structure without additional tokenizer training. At the same token count, local pooling achieves lower FID than interpolation and random-group pooling (Table 12; Q3 below). The matched-length comparisons in Table 7 further show that coarse semantics improve generation across integration mechanisms, establishing the benefit of spatial semantic abstraction beyond simply reducing the number of tokens.

Q3. Pooling neighborhood and token count.

Thank you for asking about the pooling design. We choose 2×4 local pooling to retain two-dimensional regional organization, reducing the 16×16 DINOv2 feature grid to 32 semantic tokens. The table below summarizes the token-count and pooling comparisons in Tables 6 and 12.

ComparisonRepresentationTokensFID ↓Recall ↑
GranularitySingle global token17.890.58
GranularityCoarse, local 2×4 pooling325.330.59
GranularityCoarse semantics645.220.57
GranularityDense semantic features2566.740.55
Pooling designLocal 1×8 pooling325.410.58
Pooling designLinear interpolation325.770.60
Pooling designRandom-group pooling325.780.59

Both 2×4 and 1×8 pooling work well, with FID of 5.33 and 5.41, respectively. Local 2D pooling also outperforms linear interpolation and random-group pooling.

The 32- and 64-token representations both outperform the single global token and dense 256-token features, identifying an effective intermediate granularity. Although 64 tokens achieve slightly better FID (5.22 vs. 5.33), we choose 32 as a more compact default, with half as many semantic tokens, comparable FID, and slightly higher Recall (0.59 vs. 0.57). Our finding concerns the effectiveness of coarse spatial context, not a uniquely optimal token count.

Q4. Generalization and sampling efficiency.

Thank you for requesting broader evaluation. SemPix's benefits extend beyond ImageNet-256: it improves FID on ImageNet-512 from 7.31 to 5.61 (Table 17) and also improves MS-COCO 2014 text-to-image generation (see Q5).

To test whether the gains persist with fewer solver steps, we additionally evaluate the same L-sized checkpoints at 10, 20, and 50 steps. Both models use matched Euler sampling without classifier-free guidance.

Sampling stepsPixelGen FID ↓SemPix FID ↓
1012.877.99
2010.347.46
507.535.33

SemPix outperforms PixelGen at every tested sampling budget, showing that its gains persist with fewer steps. At 50 steps, both models have the same reported inference latency of 0.32 s/image (Table 19). Classifier-free guidance distillation remains untested; we will state this limitation explicitly rather than extrapolating from the Euler results.

Q5. Stronger pixel-space baselines for text-to-image generation.

Thank you for suggesting a stronger baseline. To strengthen the comparison on MS-COCO 2014 at 256×256 resolution, we additionally train a PixelGen-style baseline and SemPix with matched REPA and perceptual supervision.

Apart from enabling these losses for the stronger-supervision pair, we retain the training and evaluation protocol in Appendix B.10. This uses 200K training steps, frozen CLIP ViT-L/14 text conditioning, 50-step Euler sampling, and a classifier-free guidance scale of 2.0.

REPA and Perceptual SupervisionGenerated Semantic StateFID ↓CLIP Cosine Similarity ↑PickScore ↑
✗✗30.470.212818.95
✗✓23.480.218219.12
✓✗5.920.246919.75
✓✓5.030.255320.91

The first pair omits REPA and perceptual losses; SemPix still uses DINOv2-derived semantic targets. The stronger pair matches pretrained visual supervision across both methods.

With these losses enabled, SemPix reduces FID from 5.92 to 5.03, a 15.0% relative reduction, while also improving CLIP cosine similarity and PickScore. This shows that generated semantic context provides additional benefits under matched, strong visual supervision in text-conditioned pixel-space generation. We will include these results in the revised manuscript.

Q6. Presentation of additional experiments and limitations.

Thank you for this presentation suggestion. We will bring the ImageNet-512 results (Table 17), MS-COCO text-to-image experiments (Table 18), and new stronger-baseline results into the main text. We will also add a dedicated limitations paragraph distinguishing our experimentally validated findings from untested extensions, including classifier-free guidance distillation and learned semantic compression.

Reviewer_XKqzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a two-track diffusion model which essentially simultaneously a coarse semantic (DINO) feature jointly with raw pixels. The basic idea is to randomly sample two independent noises for the semantic and pixel features and synchronize the forward process with the same t, thereby introducing a valid diffusion process over an overcomplete image representation (image + extracted feature).

Critical strengths

It's reasonable to use auxilliary semantic features to guide diffusion. Visual generation benefits from semantic visual features in many ways as has been shown in much prior work, but adding the semantic conditioning in this way offers the advantages of visual semantics without requiring a lossy visual tokenizer as in LDM tokenizers based on VQGAN, and doesn't add additional losses as in REPA.

Critical weaknesses

The first criticism is that there are already two extremely similar well-known prior works, namely REDI and Latent Forcing. I would expect the paper to spend considerably more time disentangling the differences between these three techniques and ablating and gaining understanding of them. Instead, they are only briefly glossed while the papers own methodology is detailed extensively. This is an overemphasis/overclaim w.r.t. novelty.

The second issue is that using DINO features on ImageNet is unfair. It is not appropriate to compare against papers such as DiT-XL/2 or the baseline in Figure 1(b), which do not leverage DINO features -- DINO being a network trained on ~140M additional images that were sourced based on ImageNet seed images, no less -- without an appropriately loud disclaimer about the advantage that the additional compute and data that DINO features provide indirectly via distillation.

Questions for authors

What are the advantages of synchronizing the noise schedule? Why not fully denoise semantics first, then pixels?

8 comments

We thank the reviewer for the positive assessment of the paper's soundness and for recognizing the value of semantic visual representations in direct pixel generation without a pretrained image autoencoder. We appreciate the questions about related work, novelty, DINO pretraining, and synchronous generation.

Q1. Differences from ReDi and Latent Forcing.

Thank you for asking us to clarify these comparisons.

Comparison with ReDi. ReDi jointly generates VAE latents and semantic features in latent space. It uses PCA to reduce the semantic feature channels and concatenates the reduced features with VAE latents along the channel dimension. SemPix studies semantic context for direct pixel-space generation. Our default design uses spatial aggregation to reduce the number of semantic tokens, then concatenates them with pixel tokens along the token dimension. This allows distinct pixel and semantic tokens to interact through joint self-attention. In Table 7, our Explicit semantic-token concatenation integration method is shown to be better than the previous Channel concatenation in Redi.

Table 7 already includes a channel-concatenation control using the same PCA-based feature reduction as ReDi. It compares semantic representations and integration mechanisms on ImageNet-256. All values are FID (↓).

Integration mechanismDense (256 tokens)Repeated coarse (256 tokens)Coarse (32 tokens)
Explicit semantic-token concatenation6.745.245.33 (Our SemPix)
Embedding addition6.875.65Incompatible
Channel concatenation (PCA-based reduction)6.62 (ReDi style)5.79Incompatible

The benefit of coarse, evolving semantic context is a generalizable finding about effective spatial semantic granularity. The advantage persists at matched token counts and across different fusion mechanisms (Tables 6–7), while Fig. 6 further reveals superior recoverability under high noise. Together, these results establish a new design principle for effective generated semantic context.

With PCA-based channel concatenation and the 256-token sequence length held fixed, coarse semantics improve FID from 6.62 to 5.79. This isolates the benefit of coarse semantic context under ReDi-style integration. With the repeated coarse representation fixed, explicit token concatenation achieves 5.24 FID, compared with 5.65 for embedding addition and 5.79 for channel concatenation. Our default uses only 32 coarse tokens and achieves 5.33 FID.

“Incompatible” indicates that embedding addition and channel concatenation require position-aligned semantic and pixel features. We will clarify the PCA implementation and configuration descriptions in the revised manuscript.

Comparison with Latent Forcing. Latent Forcing mainly studies the ordering of latent and pixel denoising through separately tuned noise schedules. They use a embedding addition integration and dual JiT heads(simply add the embedding of pixel and semantic tokens and split the last 4 JiT layers to predict different space), which is proved to be worse than our design, as shown in Table 7 and 8 in our paper. Our SemPix studies the spatial granularity of generated semantic context and its continuous use during pixel generation. We establish a new design principle for effective generated semantic context that persists across different fusion mechanisms and frameworks. In our framework, synchronous updates outperform the tested semantic-leading schedules in FID and Recall, as detailed in Q4 below.

We also compare prediction-head designs within the same SemPix backbone and training setup. The following results from Table 8 isolate the effect of target-specific refinement on ImageNet-256.

ComparisonArchitectureParametersFID ↓
Prediction headsSeparate final layers462.26M6.53
Prediction headsDual JiT heads (Latent Forcing design)464.36M6.15
Prediction headsDual refinement heads (SemPix)464.36M5.33

Replacing the dual JiT heads used in Latent Forcing with our dual refinement heads improves FID from 6.15 to 5.33 at the same parameter count level. This comparison establishes the benefit of target-specific refinement within the same generation framework.

Q2. Novelty explanation.

Thank you for asking us to clarify the novelty. Our main contribution is to establish new findings on the effective form of generated semantic context and its role throughout direct pixel-space generation. Joint image–representation generation has been explored, but these two aspects have not been systematically investigated together in this setting. Our key contributions are:

  1. Novel findings on effective semantic context. To our knowledge, we provide the first systematic evidence that coarse spatial semantics outperform global and dense alternatives as generated context in pixel-space generation (Table 6). This advantage holds at matched sequence length, across integration mechanisms (Table 7), visual encoders (Table 13), backbone architectures (Table 11), and longer training schedules (Fig. 7). The predictability analysis further shows that coarse semantic representations are easier to recover under high noise (Semantic predictability across denoising timesteps (Fig. 6)).
  2. New insights into context evolution and utilization. We show that semantic context is most effective when carried across denoising steps and directly used for pixel prediction, beyond providing semantic supervision (Table 4). Pixel–semantic consistency (Fig. 5) shows that the evolving state tracks the particular generated image beyond class conditioning. Our free-sampling analysis evaluates each model's image predictions against its own final generated image, showing earlier semantic agreement for SemPix and progressive refinement of its generated semantic state.
  3. An efficient joint pixel–semantic generation design. SemPix jointly evolves RGB pixels and coarse semantic context through flow matching. A shared Transformer enables interaction between the two states, while dual refinement heads combine the shared context with their respective noisy inputs for target-specific prediction. At a matched parameter budget, the shared backbone improves FID from 6.81 for a dual-stream model to 5.33; a full-width dual-stream model requires nearly twice the parameters to reach 5.15 FID (Table 8). The prediction-head comparison in Q1 further validates target-specific refinement.

Q3. Fairness of DINO pretraining.

Thank you for raising this fairness concern. The pixel-only baseline in Fig. 1(b) already uses pretrained DINO features. Following the PixelGen training recipe, the comparison method use DINO-based REPA alignment and perceptual supervision that includes a DINO loss (Sec. 4.1, Appendix A.1–A.3). SemPix introduces no additional pretrained visual model or external training data relative to this baseline; it makes semantic representations part of the evolving generative state.

Table 5 evaluates generated semantic context under three visual-supervision settings on ImageNet-256, each with a 200K-step training budget.

Training supervisionPixel-only FID ↓SemPix FID ↓
Neither REPA nor perceptual loss36.4528.47
REPA only23.6419.53
REPA + perceptual supervision7.535.33

The last row provides the matched-supervision comparison: SemPix improves FID from 7.53 to 5.33 under the same 200K-step budget (Tables 1 and 5). The other two rows show that the gains persist under lighter visual supervision. Even without REPA or perceptual loss, SemPix still uses DINOv2-derived semantic targets.

Following the presentation in PixelGen, Table 1 also includes DiT and PixelFlow as reference baselines. The other methods in Table 1 use REPA supervision incorporating DINO features. We will clarify this distinction in pretrained visual supervision in the revised manuscript.

Q4. Advantages of synchronous pixel–semantic generation.

Thank you for asking about the synchronous schedule. Synchronous generation maintains continuous bidirectional interaction between semantic context and evolving pixels.

In our coupled formulation (Sec. 3.1, Eqs. 2–3), the semantic state informs pixel predictions, while evolving pixels can refine subsequent semantic predictions. Fully generating semantics before pixels fixes the semantic context during pixel refinement and removes this interaction.

We would like to clarify that our SemPix already scales the semantic features by β = 0.5 to adjust their amplitude and effective SNR. In SNR terms, this corresponds to a timestep reparameterization of the unscaled representation (Appendix A.4, Eqs. 11–14).

With this feature scaling in place, we further evaluate semantic-leading schedules that explicitly offset the semantic and pixel timesteps on ImageNet-256 (Appendix B.9, Table 16). We also add a fully semantic-first experiment with δ = 1.0, which completes semantic denoising before generating pixels.

ScheduleFID ↓Recall ↑
Synchronous (SemPix)5.330.59
Semantic-leading, δ = 0.105.720.51
Semantic-leading, δ = 0.205.670.50
Semantic-leading, δ = 0.305.840.45
Semantic-leading, δ = 1.0 (fully denoise semantics first; new)7.920.35

Synchronous generation achieves the best FID and Recall across all tested schedules. Fully denoising semantics before pixels worsens FID from 5.33 to 7.92 and Recall from 0.59 to 0.35. The advantage over both partial semantic-leading and fully semantic-first schedules supports continuously refining semantic context together with pixels throughout generation.

Reviewer_wsFWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SemPix, a pixel-space diffusion model that jointly generates a small set of semantic tokens alongside the image during denoising. The goal is to give the generator an explicit semantic tokens that persists across denoising steps. During training, the semantic targets are built by locally average-pooling frozen DINOv2 patch features into 32 tokens.

Critical strengths
  1. Efficient formulation. SemPix adds only 32 semantic tokens to the pixel sequence, yet it consistently improves pixel-space generation. This low overhead makes the approach practical and easy to adopt in existing pixel-space generators.

  2. Extensive ablations. The paper systematically studies semantic granularity (global, coarse, and dense tokens), matched sequence length, integration mechanism, and more. This validates

Critical weaknesses
  1. Incremental novelty. Joint generation of visual features with images already exists in latent space as mentioned in related works. Author should explain in detail why the coarse semantic tokens in SemPix serve as better context than REG’s global token or ReDi’s dense features, beyond the empirical ablations. It would also strengthen the paper to clarify whether the approach generalizes to latent diffusion models or whether its benefits are specific to pixel space. If the latter, the authors should explain why.
  2. Missing baseline and ablation. ReDi reduces the channel dimensionality of dense features (e.g., via PCA) rather than the number of tokens. This channel-reduction alternative is not evaluated in pixel space.
  3. Motivation.
    • The paper argues that pixel-space generators lack semantic context, but it is never showed. It would strengthen the paper to include failure cases from the pixel-space baseline visually, for example semantic content becoming inconsistent across denoising steps, and how SemPix resolves the problem with the semantic tokens.
    • The work is motivated by analogy to context in LLMs, but this analogy is never revisited or tested in the later sections of the paper. The authors should either demonstrate how the semantic tokens act like LLM context, for example by showing that the semantic tokens and the denoised image exhibit a relationship similar to that between the context and subsequently generated tokens in an LLM, or tone down the LLM analogy and focus on the characteristics of pixel-space diffusion.
Questions for authors
  1. Analysis of the learned semantic tokens. The paper lacks an analysis of the learned semantic tokens. It would be valuable to examine how their representations evolve across denoising steps.
  2. Inconsistent training budgets. In Table 2 and Table 3, baselines are trained for different number of epochs, ranging from 80 to 400 and 160 to 1400. Could the authors explain the choice of these difference in training budgets and report performance at matched epochs for fair comparison?
  3. Table reference. Several results in the experiments section are discussed without reference to their corresponding tables, which makes the section difficult to follow. For example, Tables 4 and 7 are never cited in the main text. Could the authors explicitly cite the relevant tables?
10 comments

Thank you for recognizing the efficiency of our semantic-token formulation and the extensive controlled ablations. We appreciate your suggestions on the contribution, motivation, semantic analysis, and presentation of the paper.

Q1. Novelty explanation.

Thank you for raising this question. Our contribution lies in establishing what form of generated semantic context is effective and how it should be used throughout pixel generation. Prior work has explored joint image–representation generation, but these two questions have not been systematically investigated together in direct pixel-space generation. Our key contributions are:

  1. Novel findings on effective semantic context. To our knowledge, we provide the first systematic evidence that coarse spatial semantics outperform global and dense alternatives as generated context in pixel-space generation (Table 6). We control for sequence length and integration mechanism (Table 7), and verify this advantage across visual encoders (Table 13), backbone architectures (Table 11), and longer training schedules (Fig. 7).
  2. New insights into context evolution and utilization. We show that semantic context is most effective when carried across denoising steps and directly used for pixel prediction, rather than used only for semantic supervision. The interventions in Table 4 establish the importance of state persistence and semantic-to-pixel interaction. Fig. 5 further shows that the evolving state tracks the particular generated image beyond class conditioning.
  3. An efficient shared architecture with target-specific refinement. A shared Transformer allows pixel and semantic tokens to exchange information, while dual refinement heads combine the shared context with their respective noisy inputs for target-specific prediction. At a matched parameter budget, this design improves FID from 6.81 for a dual-stream backbone to 5.33; a full-width dual-stream model requires nearly twice the parameters to reach 5.15 FID (Table 8). With 32 semantic tokens, SemPix also matches PixelGen's measured inference latency of 0.32 s/image (Table 19).

Q2. Benefits of coarse semantic context and latent generalization.

Thank you for the question. Coarse semantics provide a better balance between spatial structure and prediction difficulty.

A REG-style global token captures image-level semantics but does not retain regional organization. Dense patchwise features preserve fine-grained details, but they are harder to predict at high noise. Local aggregation retains regional semantic structure while reducing this prediction burden.

Table 6 compares these representations on ImageNet-256.

Semantic representationSemantic tokensFID ↓Recall ↑
Single global token17.890.58
Coarse spatial semantics (SemPix)325.330.59
Coarse spatial semantics645.220.57
Dense semantic features2566.740.55

To examine prediction difficulty, we compare coarse and dense targets using variance-normalized MSE and cosine similarity. The controlled analysis shows that coarse targets are more reliably recovered in the high-noise regime, and this advantage largely persists when the coarse representation is expanded to 256 positions (Semantic predictability across denoising timesteps (Fig. 6)). Table 7 additionally controls for sequence length and integration mechanism, as discussed in Q3.

Regarding latent diffusion, our joint flow-matching formulation operates on coupled continuous states without requiring the image state to be RGB pixels (Eqs. 1–3). We further validate its applicability to latent-space generation by incorporating SemPix into SiT-B/2, with both models trained for 400K iterations.

Additional experiment: Latent-space generation with SiT-B/2.

MethodTraining iterationsFID ↓
SiT-B/2400K34.74
SiT-B/2 + SemPix400K26.48

SemPix reduces FID from 34.74 to 26.48 under the same training budget. This provides direct empirical evidence that generated semantic context also benefits latent-space generation, showing that its effectiveness is not restricted to pixel-space generation.

Q3. PCA-based channel reduction.

Thank you for suggesting this comparison. We would like to claify that the requested PCA-based control is already included in Table 7, but we named its implementation not as PCA but as channel-concatenation. The channel-concatenation variant uses the same PCA-based feature reduction as ReDi: we use PCA to reduce the dimension of the semantic features, concatenate the reduced features with pixel features, and apply a linear projection.

Table 7 compares semantic representations and integration mechanisms on ImageNet-256. All values are FID (↓).

Integration mechanismDense (256 tokens)Repeated coarse (256 tokens)Coarse (32 tokens)
Explicit semantic-token concatenation6.745.245.33 (SemPix)
Embedding addition6.875.65Incompatible
Channel concatenation (PCA-based reduction)6.62 (ReDi style)5.79Incompatible

This controlled comparison confirms that the benefit persists under PCA-based channel reduction and is not explained solely by fewer tokens or a different integration mechanism. Our SemPix uses explicit semantic-token concatenation with 32 coarse tokens and achieves 5.33 FID. Explicit token concatenation allows distinct pixel and semantic tokens to interact through joint self-attention and achieves the best FID among the tested integration mechanisms.

“Incompatible” indicates that embedding addition and channel concatenation require position-aligned semantic and pixel features. We will clarify the experiment configuration details in the revised manuscript.

Q4. Motivation and the role of semantic context.

Thanks for the question. The distinction is between implicit semantic information and an explicitly propagated, image-specific semantic state. The pixel-only baseline can learn semantic information, but it does not carry a separate generated semantic state across denoising steps (Sec. 1, Fig. 1(a)).

To test the role of this state, the ablations in Table 4 remove semantic carryover or block semantic-to-pixel attention. Each variant is trained and evaluated with its respective intervention; the intervention is not applied only at inference.

VariantFID ↓
Full interaction (SemPix)5.33
No semantic carryover8.30
Block semantic-to-pixel attention8.12
Downsampled pixel auxiliary state8.58

Removing semantic carryover worsens FID from 5.33 to 8.30, while blocking semantic-to-pixel attention worsens it to 8.12. These results establish the importance of carrying the state across steps and using it for pixel prediction.

The gain is also not explained by adding an auxiliary state or extra refinement capacity. Replacing the semantic state with a same-shaped downsampled pixel state gives 8.58 FID. Adding a refinement head to the pixel-only baseline likewise gives no improvement (7.53 vs. 7.62 FID; Table 10). These controls isolate the contribution of meaningful semantic context.

We also examine how this context affects generation. Appendix F (Figs. 8–9) provides qualitative trajectory comparisons, and Fig. 5 shows that the semantic state tracks the particular emerging image beyond class conditioning. Full-resolution comparisons of JiT, PixelGen with REPA and perceptual supervision, and SemPix use identical class labels and initial pixel noise:

Labrador retriever, Goldfinch, and Gazelle

We add an additional 2,000-trajectory analysis. For each method, we compare its predicted clean image at each step with its own final generated image. SemPix establishes the semantic content of its eventual image earlier and maintains stronger semantic agreement through nearly the entire trajectory. This result is consistent across DINOv2 and the independent SigLIP encoder:

Evolution of image and semantic-token predictions during free sampling

The analogy to LLM context is functional rather than architectural: carried information informs subsequent predictions. SemPix continuously updates its semantic state, whereas an autoregressive history retains earlier tokens. We will clarify this distinction in the revised manuscript.

Q5. Analysis of the learned semantic tokens.

Thank you for asking about semantic evolution. Figures 5–6 analyze instance-specific semantic correspondence and noise-dependent recoverability (Appendices B.2–B.3). We further examine how semantic-token predictions evolve during free sampling and recover clean targets under controlled noising.

Instance-specific correspondence. Fig. 5 compares predicted semantic states with DINOv2 features extracted from the corresponding predicted images. Matched trajectories show higher consistency than independent same-class or different-class pairs, demonstrating semantic evolution beyond class conditioning. This analysis covers 2,000 trajectories across 1,000 classes, with bootstrap 95% confidence intervals:

Instance-specific pixel–semantic consistency across denoising (Fig. 5)

Formation and refinement during free sampling. We additionally examine semantic-token predictions over 2,000 free-sampling trajectories. The clean-token predictions progressively approach their final generated semantic state: cosine similarity increases and MSE decreases. This directly characterizes how generated semantic context forms and is refined during sampling:

Evolution of image and semantic-token predictions during free sampling

Recovery under controlled noising. In a separate analysis of 2,000 samples, we add noise to real images and their semantic targets and evaluate clean-target predictions at different noise levels. SemPix progressively recovers the original semantic tokens as noise decreases. Its clean-image predictions also show stronger semantic agreement with the original images than pixel-only baselines under the corresponding joint-denoising setting, consistently in both DINOv2 and SigLIP feature spaces:

Clean image and semantic-token recovery under controlled noising

The coarse-versus-dense analysis in Fig. 6, discussed in Q2, further measures noise-dependent semantic recoverability. These analyses jointly show how semantic context evolves during generation and how reliably the model predicts meaningful representations across noise levels.

Q6. Inconsistent training budgets.

Thank you for raising this concern. Tables 2–3 report published baselines at their original training budgets and include multiple SemPix checkpoints. The former provide broader performance references, while the latter show training convergence; the cross-paper comparisons are not intended to be fully compute-matched.

SemPix-XL reaches 4.93 FID after 40 epochs, already outperforming PixelGen-XL's 5.11 FID after 80 epochs (Table 2). The advantage also holds at matched epochs: 3.83 vs. 5.11 FID at 80 epochs (Table 2), and 1.72 vs. 1.83 at 160 epochs (Table 3).

Table 1 further provides a controlled 200K-step comparison, where SemPix improves FID from 7.53 to 5.33. These results support both faster convergence and improved generation quality under matched training budgets.

Q7. Table references.

Thank you for pointing out the missing references. We will cite Table 4 in Sec. 4.3 when discussing semantic-state persistence and interaction, and Table 7 in Sec. 4.4 when discussing matched sequence length and integration mechanisms.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Author Rebuttal

Dear Area Chair and all Reviewers,

We sincerely thank all reviewers for their thorough, constructive feedback, which has greatly helped us improve the manuscript and strengthen empirical evidence. We thank the reviewers for recognizing our method formulation, extensive and carefully controlled ablations, the value of our method design and semantic advantage, performance improvements, and clear explanation.

We want to briefly follow up during the discussion period and make sure that the new evidence corresponding to their questions is easy to locate.

For Reviewer wsFW, our response now includes: (1) Explanation of our unique novelty points that previous works haven't addressed and explanation in detail why the coarse semantic tokens are better; (2) An additional latent space experiment demonstrating that our method also generalizes to latent diffusion models and its benefits are not specific to pixel space; (3) Clearer explanation of the requested ReDi style channel-reduction comparison, which was already included in Table 7 in our paper. We reorganized Table 7, our results show that not only our SemPix style token concatenation is better, but also our novel findings that "coarse semantics serves better context than the original dense semantics (ReDi style) " persist on channel concat integration ; (4) Additional analysis of the semantic tokens and pixel tokens showing that our semantic are consistently better and semantic tokens are consist with the pixel tokens. Failure cases from the pixel-space was already included in Appendix F (Figs. 8–9) in our paper; (5) A revised and more precise wording of training budgets and table references.

For Reviewer XKqz, our response now includes: (1) Detailed explanation about the differences with ReDi and Latent Forcing which lies in diffusion space, integration mechanism, model architecture and espectially insight findings. We clarify that ReDi style Channel concatenation and and Latent Forcing style Embedding addition experiment is already in Table 7 in our paper. And model backbone design and prediction-head comparisons were already included in Tables 8 in our paper. (2) Explanation of our unique novelty points that previous works haven't addressed and explanation in detail our effective form of semantic context and how it is carried and used throughout pixel generation; (3) Explanation of the fairness of DINO features. We clarify that in Table 5 abltion in our paper, we already conducted a detailed breakdown of this . We distinguish that the main comparison methods have already incorporated REPA and perceptual losswhich already contains DINO supervision. Moreover, we generate coarse DINO features simultaneously as context to assist pixel generation, which need no input DINO features; (4) An additional experiment that fully denoises semantics first, then pixels, answering the question about the advantages of synchronizing the noise schedule. Synchronous generation achieves better FID and Recall in this comparison.

For Reviewer 7VnH, our response now includes: (1) Detailed explanation of our novelty and how SemPix differs from Representation Forcing in its formulation, architecture and use of the semantic state. And a clearer explanation of our unique novelty points that previous works haven't addressed; (2) Clearer explanation to relative semantic tokenizers and token-reduction methods like SEED and BEiT v2/VQ-KD; (3) Explanation of the pooling design and the neighborhood shape and token count comparisons, which are already included in our paper in Tables 6 and 12; (4) An additional comparison of fewer sampling steps for PixelGen and SemPix, showing that the gains persist with fewer sampling steps; (5) An additional COCO text-to-image experiment showing that our method achieves better FID against a stronger pixel-space baseline with REPA and perceptual supervision; (6) A revised and more precise wording of ImageNet-512 and text-to-image results and their limitations, and a commitment to to include them in the main text.

We provide detailed point-by-point responses to each reviewer's comments below, and we hope our responses fully address all concerns, and we look forward to further discussing the paper with you. Should you request additional experiments or analyses, please do not hesitate to let us know. We will do our best to deliver results promptly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.