- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SemPix, a pixel-space image generation model that jointly generates pixels and a compact set of semantic tokens. The semantic tokens are carried across denoising steps and provide context for subsequent pixel predictions. Their training targets are obtained by applying parameter-free average pooling to frozen DINOv2 features. Pooling a 16×16 feature grid over 2×4 neighborhoods produces 32 continuous tokens that retain spatial structure. The pixel and semantic tokens are processed by a shared Transformer, with separate refinement heads predicting the two targets. No visual encoder is needed at inference. On class-conditional ImageNet-256, SemPix-XL/2 achieves an FID of 1.72 and an IS of 316.1 with classifier-free guidance after 160 epochs. At a matched 40-epoch budget, SemPix-L/2 improves FID from 7.53 to 5.33 relative to PixelGen-L/2, with essentially unchanged reported per-step time and memory usage. The ablations examine both the choice of semantic representation and how it is used during sampling. Additional results on ImageNet-512 and text-to-image generation are provided in the appendix.
Critical strengths
1.The ablations are carefully controlled. The matched-length comparisons help separate the benefit of coarse semantic representations from that of using fewer tokens. The interventions in Table 4 also test whether the semantic state needs to persist across steps and whether pixel tokens need access to it. These experiments provide useful evidence for the proposed mechanism.
2.The finding that coarse, spatially organized semantic tokens work better than either dense features or a single global token is interesting. The paper tests this choice with different encoders and integration mechanisms, which makes the observation more convincing than a result from one configuration.
3.The improvements come with little additional per-step overhead in the reported comparison. The reduction from 7.53 to 5.33 FID at the same training budget is substantial. The XL model also reaches competitive results with 687M parameters, compared with 918M for V-Co-XL/2. The absence of an external visual encoder at inference is another useful practical property.
4.The paper is generally clear and easy to follow. The architecture and experimental motivation are explained well, and the positioning table in Appendix E helps distinguish the method from REG, ReDi, and V-Co.
Critical weaknesses
1.My main concern is whether the paper establishes its conceptual novelty clearly enough. Representation Forcing appears closely related at a high level: its abstract describes using predicted visual representations as context for pixel-space generation without a VAE bottleneck. However, the paper does not cite or discuss it. The comparison with REG, ReDi, V-Co, and Latent Forcing is helpful, but I would still like the authors to explain how SemPix differs from Representation Forcing in its formulation, architecture, and use of the semantic state. Without that discussion, it is difficult to judge how much of the contribution comes from a new generation mechanism and how much comes from the particular representation and implementation choices.
2.The construction of the 32 semantic tokens is simple average pooling, so the novelty does not appear to lie in the compression operation itself. The related-work discussion would benefit from a clearer connection to semantic tokenizers and token-reduction methods. For example, SEED also compresses a 16×16 feature grid into 32 tokens for generation, although its learned, causal, quantized representation differs substantially from SemPix’s continuous spatial tokens. Work such as BEiT v2/VQ-KD is also relevant to the use of teacher features as semantic targets. These connections do not negate the paper’s empirical findings, but discussing them would help identify the specific contribution more precisely.
3.The experiments support the method within the settings studied, but the broader claims about semantic context are less well established. Most of the evidence comes from class-conditional ImageNet-256 with a fixed 50-step sampler. There is no evaluation with fewer sampling steps or after distillation. The COCO text-to-image experiment is useful, but it only compares against a pixel-only baseline. I would prefer the conclusions to distinguish the findings demonstrated by these experiments from the broader motivation for the method.
4.There are a few minor presentation issues. The limitations are spread across the reproducibility statement and appendix discussion, which makes them harder to locate. The ImageNet-512 and text-to-image results are also confined to the appendix. A brief discussion of these results and their limitations in the main text would give a more complete picture of the method’s scope.
Questions for authors
1.How does SemPix differ from Representation Forcing in the way visual representations are predicted and used to guide pixel generation? Please discuss the relationship and, if feasible, include an empirical comparison.
2.How should the proposed tokens be understood relative to SEED-style semantic tokenizers and existing token-reduction methods? Is parameter-free average pooling important to the observed gains, or might a learned compressor or quantized representation work equally well?
3.Why was an asymmetric 2×4 pooling neighborhood chosen? How sensitive are the results to the neighborhood shape and token count? In particular, Table 6 suggests that 64 tokens perform slightly better than 32.
4.Do the gains persist with fewer sampling steps or after classifier-free guidance distillation? A comparison of FID against sampling steps for PixelGen and SemPix would help answer this.
5.Can the text-to-image experiment include stronger pixel-space baselines? This would make it easier to assess the significance of the COCO result.
11 comments
Q3. Pooling neighborhood and token count.
Thank you for asking about the pooling design. We choose 2×4 local pooling to retain two-dimensional regional organization, reducing the 16×16 DINOv2 feature grid to 32 semantic tokens. The table below summarizes the token-count and pooling comparisons in Tables 6 and 12.
| Comparison | Representation | Tokens | FID ↓ | Recall ↑ |
|---|---|---|---|---|
| Granularity | Single global token | 1 | 7.89 | 0.58 |
| Granularity | Coarse, local 2×4 pooling | 32 | 5.33 | 0.59 |
| Granularity | Coarse semantics | 64 | 5.22 | 0.57 |
| Granularity | Dense semantic features | 256 | 6.74 | 0.55 |
| Pooling design | Local 1×8 pooling | 32 | 5.41 | 0.58 |
| Pooling design | Linear interpolation | 32 | 5.77 | 0.60 |
| Pooling design | Random-group pooling | 32 | 5.78 | 0.59 |
Both 2×4 and 1×8 pooling work well, with FID of 5.33 and 5.41, respectively. Local 2D pooling also outperforms linear interpolation and random-group pooling.
The 32- and 64-token representations both outperform the single global token and dense 256-token features, identifying an effective intermediate granularity. Although 64 tokens achieve slightly better FID (5.22 vs. 5.33), we choose 32 as a more compact default, with half as many semantic tokens, comparable FID, and slightly higher Recall (0.59 vs. 0.57). Our finding concerns the effectiveness of coarse spatial context, not a uniquely optimal token count.
Q4. Generalization and sampling efficiency.
Thank you for requesting broader evaluation. SemPix's benefits extend beyond ImageNet-256: it improves FID on ImageNet-512 from 7.31 to 5.61 (Table 17) and also improves MS-COCO 2014 text-to-image generation (see Q5).
To test whether the gains persist with fewer solver steps, we additionally evaluate the same L-sized checkpoints at 10, 20, and 50 steps. Both models use matched Euler sampling without classifier-free guidance.
| Sampling steps | PixelGen FID ↓ | SemPix FID ↓ |
|---|---|---|
| 10 | 12.87 | 7.99 |
| 20 | 10.34 | 7.46 |
| 50 | 7.53 | 5.33 |
SemPix outperforms PixelGen at every tested sampling budget, showing that its gains persist with fewer steps. At 50 steps, both models have the same reported inference latency of 0.32 s/image (Table 19). Classifier-free guidance distillation remains untested; we will state this limitation explicitly rather than extrapolating from the Euler results.
Q5. Stronger pixel-space baselines for text-to-image generation.
Thank you for suggesting a stronger baseline. To strengthen the comparison on MS-COCO 2014 at 256×256 resolution, we additionally train a PixelGen-style baseline and SemPix with matched REPA and perceptual supervision.
Apart from enabling these losses for the stronger-supervision pair, we retain the training and evaluation protocol in Appendix B.10. This uses 200K training steps, frozen CLIP ViT-L/14 text conditioning, 50-step Euler sampling, and a classifier-free guidance scale of 2.0.
| REPA and Perceptual Supervision | Generated Semantic State | FID ↓ | CLIP Cosine Similarity ↑ | PickScore ↑ |
|---|---|---|---|---|
| ✗ | ✗ | 30.47 | 0.2128 | 18.95 |
| ✗ | ✓ | 23.48 | 0.2182 | 19.12 |
| ✓ | ✗ | 5.92 | 0.2469 | 19.75 |
| ✓ | ✓ | 5.03 | 0.2553 | 20.91 |
The first pair omits REPA and perceptual losses; SemPix still uses DINOv2-derived semantic targets. The stronger pair matches pretrained visual supervision across both methods.
With these losses enabled, SemPix reduces FID from 5.92 to 5.03, a 15.0% relative reduction, while also improving CLIP cosine similarity and PickScore. This shows that generated semantic context provides additional benefits under matched, strong visual supervision in text-conditioned pixel-space generation. We will include these results in the revised manuscript.
Q6. Presentation of additional experiments and limitations.
Thank you for this presentation suggestion. We will bring the ImageNet-512 results (Table 17), MS-COCO text-to-image experiments (Table 18), and new stronger-baseline results into the main text. We will also add a dedicated limitations paragraph distinguishing our experimentally validated findings from untested extensions, including classifier-free guidance distillation and learned semantic compression.
We thank the reviewer for recognizing our controlled ablations, the effectiveness of coarse spatial semantic context, and the low measured computational overhead. We address the questions below, including new experiments with fewer sampling steps and stronger MS-COCO text-to-image baselines.
Q1. Relationship to Representation Forcing (RF).
Thank you for highlighting this related work. Both methods use generated visual representations for pixel generation. RF autoregressively predicts discrete visual representation tokens before pixel diffusion and uses the completed representations as fixed context. SemPix jointly generates continuous semantic states and RGB pixels from noise through flow matching, enabling bidirectional refinement throughout generation.
The architectures also differ. RF builds on a pretrained Qwen3 Mixture-of-Experts language model with a Mixture-of-Transformers architecture, a jointly trained DINOv3 understanding encoder, and online vector quantization. SemPix uses a shared generative Transformer with dual refinement heads and semantic targets from frozen DINOv2 (Sec. 3, Table 8).
These designs serve different research goals: RF targets unified multimodal understanding and generation through an autoregressive framework, whereas SemPix investigates how evolving semantic representations serve as context for direct pixel-space generation. Our work establishes two key findings that RF does not systematically investigate:
Novel findings on effective semantic context. RF uses 2×2 spatial pooling without systematically studying semantic granularity. We demonstrate that coarse spatial semantics outperform global and dense representations as generated context (Table 6).
This advantage holds at matched sequence length, across integration mechanisms (Table 7), visual encoders (Table 13), backbone architectures (Table 11), and longer training schedules (Fig. 7). Coarse semantic representations are also easier to recover under high noise (Semantic predictability across denoising timesteps (Fig. 6)).
New insights into context evolution and utilization. We show that semantic context is most effective when carried across denoising steps and directly used for pixel prediction. Table 4 validates the importance of both cross-step persistence and semantic-to-pixel interaction.
Pixel–semantic consistency (Fig. 5) shows that the evolving state tracks the particular generated image beyond class conditioning. Our quantitative Evolution of image and semantic-token predictions during free sampling tracks image and semantic-token predictions, showing earlier agreement with each model's eventual generated image and progressive refinement of SemPix's semantic state.
We will clarify these distinctions and discuss RF in the revised manuscript.
Q2. Semantic token construction and relation to existing tokenizers.
Thank you for suggesting this tokenizer comparison. Our contribution is the generation and use of continuously evolving semantic context in pixel-space generation, rather than the average-pooling operator itself. We systematically study the effective form of this context and its role throughout the generation trajectory (Tables 4, 6–7).
SEED learns 32 quantized, causally ordered visual tokens for autoregressive multimodal generation; SemPix instead maintains continuous, spatially organized semantic states that are jointly generated and refined with pixels. BEiT v2/VQ-KD discretizes teacher features as targets for masked image modeling, rather than carrying them as evolving generative states.
We use parameter-free local pooling to preserve regional semantic structure without additional tokenizer training. At the same token count, local pooling achieves lower FID than interpolation and random-group pooling (Table 12; Q3 below). The matched-length comparisons in Table 7 further show that coarse semantics improve generation across integration mechanisms, establishing the benefit of spatial semantic abstraction beyond simply reducing the number of tokens.