acceptodds
Under review as a conference paper at ICLR 2027

SemPix: Learning Semantic Flows for Pixel Generation

14 views

Abstract

Pixel diffusion generates images directly in pixel space without a separately trained autoencoder. Yet a single pixel trajectory must model object structure and spatial layout together with the diverse textures, colors, and illumination that realize them. Pretrained visual representations reduce sensitivity to such appearance variation and organize images by content and spatial structure, providing a more direct target for the visual structure required by pixel generation. We introduce SemPix, a coupled semantic–pixel flow framework that models pretrained visual representations as an explicit generative state for native pixel synthesis, rather than as auxiliary supervision alone. A semantic flow learns the distribution of dense pretrained features, while a pixel flow generates the complete image from independent noise under spatially aligned modulation by the generated state. This coupling makes semantic structure part of the generative process while preserving stochastic generation in native pixel space. The two flows are trained end-to-end through predicted semantic states, combining representation supervision with pixel-generation gradients that adapt the semantic state for image synthesis. On class-conditional ImageNet 256×256, SemPix converges substantially faster than native pixel diffusion baselines without guidance and achieves competitive generation quality against state-of-the-art pixel generators after full training. These results show that modeling semantic structure as an explicit generative state can reduce the optimization burden of native pixel diffusion.

Primary area:
generative models
Keywords:
Pixel Diffusion, Image Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vUHVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the accelerating training technique for pixel diffusion. It proposes SemPix, which generates a semantic state in the frozen DINOv3 feature space with a semantic flow and conditions a shallow pixel flow on it. The claimed contribution is coupled training, allowing image-loss gradients to flow back into the semantic flow through the predicted condition.

Critical strengths
  1. It is a meaningful topic to accelerate pixel diffusion training, and decoupling it into semantic and pixel components is reasonable.
Critical weaknesses
  1. The convergence comparison is confounded by LPIPS loss and other loss components. Every SemPix run includes Lperc{L}_{{perc}} in Eq. 7, while the baselines plotted in Fig. 4 do not, so the paper cannot distinguish its method from a known perceptual-loss effect.

  2. The paper's own ablations do not support the claimed contribution. The paper's core contribution is CT, which it presents as gradient feedback. However, as shown in Tab. 3b, it only brings a 0.10.1 FID improvement, which falls within the noise interval. Detailed ablations of each key component, including  Lmid{L}_{{mid}}, are necessary.

  3. Missing baselines. In Tab. 2, RAE, PiD, RAEv2, PixelDiT, and CoReDi are missing. Moreover, why are JiT+REPA and DeCo compared in Fig. 4 but not reported in Tab. 2?

  4. Some potential inconsistency in the results.

  • L320-321 claims "Table 1...SemPix achieves...IS 195.0", but Tab.1 reports IS 195.2.
  • Tab.1 reports PixNerd-L/16 has params of 459M, but in Tab.2 it becomes 458M.
Questions for authors
  1. Can the authors provide Tab. 3 rows without LpercL_{{perc}} and without Lmid{L}_{{mid}}, and an oracle FID that conditions the pixel flow on real encoder features?

  2. Since Eq. 2 and 4 are loss-equivalent to the velocity objective, what justifies xx-prediction for the semantic flow over vv-prediction?

  3. This paper claims faster convergence on unguided scenarios. How does it perform in guided scenarios, which are more practical?

8 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the detailed and technical review and the constructive suggestions. Following them, we conducted new experiments, summarized below:

  • Perceptual loss (W1, Q1). The gain from the semantic flow holds without Lperc\mathcal{L}_{\text{perc}}, which acts on the rendering quality of the pixel flow (loss-matched comparisons, ablations and oracle FIDs).
  • CT (W2). The gain is consistent across all 18 runs (3 training × 6 sampling seeds). Cross-pairing shows that it improves the semantic flow itself, and the oracle FID shows that it adapts the pixel flow to the semantic flow's predictions rather than to real encoder features.
  • Baselines (W3). We compare with RAE, RAEv2, CoReDi, PixelDiT, DeCo, PixelGen and V-Co.
  • Guided setting (Q3). At 40 epochs SemPix-L/16 reaches 2.99 with guidance, vs. 4.71 for DeCo-L/16, 4.95 for PixelDiT-L and 8.92 for PixelREPA-L/16.

We are particularly grateful for the suggestion of oracle FID. Because SemPix splits generation into a semantic flow and a pixel flow, this diagnostic lets us localize where each loss acts, and it has substantially strengthened our analysis.

W1 & Q1: Perceptual loss, loss ablations, and oracle FID

We thank the reviewer for raising this point. Following this comment, we added loss-matched comparisons, the ablation rows requested in Q1, and the corresponding oracle FIDs. The results show that the gain from the semantic flow holds with or without Lperc\mathcal{L}_{\text{perc}}, which acts on the rendering quality of the pixel flow.

(a) Loss-matched comparison with a single-flow model (40 epochs, unguided FID):

w/o Lperc\mathcal{L}_{\text{perc}}w/ Lperc\mathcal{L}_{\text{perc}}
JiT-L/16 + REPA (single flow)23.6710.00
SemPix-L/1615.83†3.87

† Without Lperc\mathcal{L}_{\text{perc}}, the pixel flow relies more on the low-noise end of the trajectory to recover fine detail, so we allocate more steps there: 25 pixel steps with a time shift of 0.75 (default: 10 steps, shift 10), and 25 semantic steps to keep the total at 50. The same applies to † in (b).

The single-flow numbers are from the PixelGen ablation, and within each column the pixel-space losses match. Without any perceptual loss, adding the semantic flow already lowers FID from 23.67 to 15.83, so the gain does not depend on Lperc\mathcal{L}_{\text{perc}}. The higher FID in this column comes from the pixel side: without perceptual supervision, the pixel flow renders texture and fine detail less well even from real features (oracle FID ≈ 12.4). The semantic flow trained without Lperc\mathcal{L}_{\text{perc}} remains strong: paired with a pixel flow trained with Lperc\mathcal{L}_{\text{perc}}, it reaches 4.05.

(b) Where each loss acts: ablations with oracle FID (Q1). Oracle FID conditions the pixel flow on real DINOv3 features, bypassing the semantic flow, and thus measures the pixel flow alone.

Lperc\mathcal{L}_{\text{perc}}Lmid\mathcal{L}_{\text{mid}}FIDOracle FID
✓✓3.873.24
✓4.293.21
✓15.83†12.44†
17.54†12.32†
  • Without Lperc\mathcal{L}_{\text{perc}}, the pixel flow is the bottleneck. Even when conditioned on real features it only reaches about 12.4, vs. about 3.2 with Lperc\mathcal{L}_{\text{perc}}, so better semantic states cannot fully show up in the final FID.
  • The semantic prior is largely independent of Lperc\mathcal{L}_{\text{perc}}. The cross-pairing in (a) gives 4.05, vs. 3.87 for the full model. The remaining 0.18 is the part of Lperc\mathcal{L}_{\text{perc}} that reaches the semantic flow through CT's gradient path, the same feedback effect quantified in W2.
  • Lperc\mathcal{L}_{\text{perc}} therefore improves the pixel side, not the semantic prior. Once the pixel side is no longer the bottleneck, better semantic states translate fully into lower FID. In the full model, replacing real features with sampled states raises FID only from 3.24 to 3.87: at 40 epochs the semantic flow already provides conditions close to real features, consistent with the faster convergence.
  • Lmid\mathcal{L}_{\text{mid}} acts on the semantic flow, as designed. It leaves oracle FID unchanged (3.21 vs. 3.24) and improves sampled FID (4.29 → 3.87), narrowing the gap between the two from 1.08 to 0.63.

(c) Convergence under matched losses. Without a perceptual loss on either side, SemPix at 40 epochs (15.83) is already below JiT + REPA at 80 epochs (18.60, Fig. 4 of the paper). With perceptual supervision on both sides, Table 1 of the paper already contains the comparison: 3.87 for SemPix-L/16 vs. 7.53 for PixelGen-L/16 at 40 epochs, while the larger PixelGen-XL/16 (676M) reaches 5.11 at 80 epochs. Against either class of baseline, SemPix reaches lower FID with less training.

AuthorsOpenReview ↗

Official Comment by Authors

W2: Is the CT gain within noise? Ablations of each component

We thank the reviewer for raising the question of statistical significance. To test whether the 0.1 FID difference lies within the noise range, we repeated the comparison in Table 3(b) of the paper with two more training seeds (three in total) and evaluated each model with six sampling seeds, i.e., 18 evaluations per setting. Gradient feedback gives lower FID in all 18 paired comparisons (mean 4.26 vs. 4.37; paired difference 0.11 ± 0.03, mean ± s.d. over the three training seeds). The gain is consistent across seeds and is not explained by noise.

This comment also helps us clarify the role of CT. The primary contribution of SemPix is generating an explicit semantic state and synthesizing pixels from it with a dedicated pixel flow, trained jointly with the semantic flow. The two-flow design accounts for most of the gain: with the same Lperc\mathcal{L}_{\text{perc}}, the single-flow model reaches 10.00, while two independently trained flows already reach 4.56. CT is the training scheme that couples the two flows, and the two analyses below show what it changes in each flow. They use the protocol of Table 3 in the paper (L-size, 40 epochs, unguided, no Lmid\mathcal{L}_{\text{mid}}).

(a) CT adapts the pixel flow to the semantic flow's predictions. To examine this, we compare the oracle FID, obtained by conditioning the pixel flow on real encoder features (the analogue of rFID in latent methods), with the standard FID of sampled images (the analogue of gFID).

Oracle FIDFID
Independent training2.724.56
+ pixel flow trained on predicted states (detached)3.124.39
+ gradient feedback (CT)3.214.29

Although SemPix generates in pixel space, the independently trained pixel flow shows a reconstruction–generation trade-off akin to that known from latent diffusion (e.g., VA-VAE; Yao et al., 2025): it reconstructs best (oracle FID 2.72) but generates worst (4.56), because it is trained on real encoder features yet receives generated states at inference. Noise augmentation, which our independent baseline already uses (Table 3(d) of the paper), only approximates this gap. CT addresses it more directly: the pixel flow is trained on the semantic flow's predictions (4.56 → 4.39), and the semantic flow adapts to the pixel flow through image-loss gradients (4.39 → 4.29). The slightly higher oracle FID does not affect generation: it measures the pixel flow on real encoder features, a setting that never occurs at inference, where the pixel flow is always conditioned on states produced by the semantic flow.

(b) The feedback improves the semantic flow itself. Table 3(b) of the paper compares two complete models, so it cannot tell which flow the 0.10 comes from. We cross-pair them at sampling time, taking the semantic flow from one and the pixel flow from the other. In the CT model, image-loss gradients reach the semantic flow; in the detached model, the condition is detached, so they do not. Both pixel flows are trained on predicted states in the same way.

Semantic flow ↓ / Pixel flow →CT modelDetached model
CT model4.294.27
Detached model4.484.39

Within each column the pixel flow is fixed, and in both columns the semantic flow from the CT model is better (by 0.19 and 0.12). Trained with a feature-space loss alone, the semantic flow cannot tell which deviations matter for the image; the image-loss gradient supplies that signal. Notably, the semantic flow from the CT model helps even a pixel flow it was never trained with: paired with the detached model's pixel flow, it reaches 4.27, on par with its own jointly trained pixel flow (4.29). If the feedback merely co-adapted the two flows, this gain would disappear once the pixel flow is swapped. Instead, the feedback improves the sampled states themselves, and since both semantic flows target the same frozen DINOv3 space, the improvement transfers to another pixel flow. The semantic-side gain is no smaller than the overall gap in Table 3(b) (0.10), and the pattern holds across several sampling seeds.

Ablations of the individual loss terms, including Lmid\mathcal{L}_{\text{mid}}, are in our response to W1 & Q1; Table 3(c, d) of the paper covers initialization and condition noise.

AuthorsOpenReview ↗

Official Comment by Authors

W3: Missing baselines

We thank the reviewer for these suggestions. The table below adds all the mentioned methods, as well as further pixel-space baselines: DeCo-XL/16, PixelREPA (representation alignment on JiT), PixelGen-XL/16 and V-Co (ImageNet 256×256; "–": not reported; the second term in Parameters is the separately trained decoder).

MethodParametersEpochsUnguided FID↓ / IS↑Guided FID↓ / IS↑
Latent Space
RAE, DiT$^\text{DH}$-XL839M + 415M8001.51 / 242.91.13 / 262.6
RAEv2, DiT$^\text{DH}$-XL839M + 415M801.65 / 228.01.06 / 255.3
CoReDi, SiT-XL/2675M + 50M4003.3 / 176.81.58 / 297.2
Pixel Space
DeCo-XL/16682M32014.88 / 88.21.90 / 303
PixelDiT-XL797M80–2.36 / 282.3
PixelDiT-XL797M320–1.61 / 292.7
PixelGen-XL/16676M805.11 / 159.2–
PixelGen-XL/16676M160–1.83 / 293.6
PixelREPA-L/16459M600–2.11 / 309.5
PixelREPA-H/16953M600–1.81 / 317.2
V-Co-L/16918M200–2.10 / 243.0
V-Co-H/161.9B200–1.85 / 246.5
SemPix-L/16471M803.27 / 202.92.17 / 245.0
SemPix-H/16970M1602.40 / 225.21.83 / 250.4

Among pixel-space methods, SemPix reaches lower FID with less training. Without guidance, SemPix-L/16 reaches 3.27 at 80 epochs, compared with 5.11 for PixelGen-XL/16 at the same budget and 14.88 for DeCo-XL/16 at 320 epochs. With guidance, it reaches 2.17 at 80 epochs, below PixelDiT-XL at the same budget (2.36) with fewer parameters. SemPix-H/16 reaches 1.83 at 160 epochs, comparable to models trained considerably longer (V-Co-H/16: 1.85 at 200 epochs; PixelREPA-H/16: 1.81 at 600 epochs).

RAE and RAEv2 generate in a representation latent space and reconstruct images with a separately trained ViT-XL decoder, whereas SemPix has no separate decoder stage. PiD, which we discuss in the related work of the paper, is not included in the table, as it evaluates latent decoding and upsampling for text-to-image models and reports no class-conditional ImageNet results. For CoReDi, the latent-space model is listed above; its pixel-space model reports only an unguided result at 200K steps (FID 21.5), which is already in Table 1 of the paper.

Reviewer_EYusOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This work introduces SemPix, an image generation framework that first generates with a flow-matching model the semantic representations (e.g., DINOv3) of an image and then, conditioned on these generated semantic representations, it generates with another flow matching model the pixels of this image. The representation-space flow matching model, called Semantic Flow, is more "heavy-weight" and essentially follows the same design as in RAE (v2): DDT architecture, x-prediction with v-loss, and an additional REPA loss (these are the same design choices as in RAEv2). The pixel-space flow matching model, called Pixel Flow, is a more "light-weight" DiT model conditioned on the semantic representations through patch-wise AdaLN layers, and trained with both a pixel-space flow-matching loss and a perceptual loss, as in PixelGen.

During training, first, as a warm-up, the two flow-matching modules are independently trained with the Pixel Flow model getting as condition the "ground-truth" semantic features with some added noise. After this warm-up, the training is now "coupled": the Semantic Flow model predicts semantic features with a single forward pass, which are given as input to the Pixel Flow model. Gradients are backpropagated from the Pixel Flow model to the Semantic Flow model.

The proposed approach is evaluated on class-conditioned image generation on ImageNet at 256 x 256 resolution. They show faster training convergence and better results than prior pixel-space flow-matching methods.

Critical strengths

In my view the main strength of this work is that, compared to the RAE works, it implements the decoding from semantic representations to image pixels with a light-weight flow-matching model that is jointly trained with the RAE-style representation-space flow-matching model. Those seem interesting design choices. However, as I discuss below, the work does not clearly position itself w.r.t. prior work and RAE, and lacks comparisons with RAE.

Critical weaknesses
  • The main weakness of this work is that it lacks a clear positioning w.r.t. the most relevant prior works, namely RAE and RAE v2. As far as I understood, the main difference of SemPix with RAE is that in SemPix the representation-to-pixel decoder is implemented with a flow-matching model that is trained jointly with the RAE-style flow-matching model that generates the semantic representations. From the ablations in Table 3, this coupled / joint training seems to have relatively small impact (4.56 FID vs 4.29 FID). Also, formulating the decoding from latents to image pixels as a diffusion problem has has already be done in "Diffusion autoencoders are scalable image tokenizers" (https://arxiv.org/abs/2501.18593). Therefore, it is not clear what is the actual technical contribution / novelty of this work.

  • The work lacks comparisons with RAE and RAE v2 methods, both at the system-level as well as controlled apples-to-apples comparisons that isolate the main factors that change between the two methods (e.g., representation-to-pixel decoder design, training strategy)? Therefore, it is not clear if the proposed framework has any benefit over RAE (v2).

Because of the above two reasons, I scored the work low on informativeness.

Questions for authors
  • Clearly discuss the position and novelty of this work w.r.t. RAE (v2).
  • Compare with RAE, both at the system level and with controlled apples-to-apples comparisons.
  • The work also lacks citing and comparing with the state-of-the-art joint pixel-feature diffusion method "V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising" (https://arxiv.org/abs/2603.16792). Is SemPix better than this joint method?
9 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the careful summary of our method and for recognizing the lightweight, jointly trained pixel flow as an interesting design. We also thank the reviewer for the suggestion to position SemPix more clearly with respect to RAE and RAEv2. Since Q1 and Q2 are closely related, we answer them together: we first identify where SemPix departs from RAE, then verify each difference with controlled comparisons. In brief:

  • (i) The pixel side is a generator, not a reconstruction decoder. With everything else fixed, sampling the pixel side instead of decoding it in one pass lowers FID from 6.91 to 4.56.
  • (ii) Training on the prior's predictions mitigates the train–inference mismatch. In two-stage pipelines such as RAE, the decoder is trained on real encoder features but receives generated states at inference, a gap the RAE paper itself reports. Training the pixel side on the semantic flow's predictions lowers FID to 4.39.
  • (iii) Gradient feedback improves the semantic flow itself. Feeding image-loss gradients back lowers FID further to 4.29, and cross-pairing the two flows shows that this gain comes from the semantic flow.
  • V-Co (Q3). At a similar or shorter training budget, SemPix matches it with about half the parameters.

Q1 & Q2: Relation to RAE (v2), with controlled comparisons

Where SemPix departs from RAE. Both RAE and SemPix first generate a state in a pretrained representation space; they differ in how the image is produced from that state and in how the two parts are trained:

Pixel sidePixel side is trained onPrior and pixel side
RAE / RAEv2deterministic ViT decoder (L1, LPIPS, GAN)noised encoder featurestrained separately
DiTodiffusion decoderlatents of an encoder trained jointly with itlatent generator trained afterwards
SemPixpixel flow, sampled from independent noisethe semantic flow's predictionsone run; image-loss gradients reach the semantic flow

In RAE, the decoder is trained to reconstruct images from encoder features and is fixed before the generator is trained. SemPix changes this pipeline in three ways, (i)–(iii) below. All controlled comparisons fix the architecture, encoder, model size (L), training budget (40 epochs) and the 40-step semantic sampling, and report unguided FID; as in Table 3 of the paper, Lmid\mathcal{L}_{\text{mid}} is not used. DiTo, which the reviewer mentions, also uses a diffusion decoder, but like RAE trains it for reconstruction from encoder latents, separately from the generator, so it differs from SemPix in (ii) and (iii).

(i) The pixel side is a generator, not a reconstruction decoder. An RAE decoder maps one latent to one image. SemPix's pixel flow models the distribution of images compatible with a semantic state, so appearance left unspecified by the state is sampled rather than regressed. To isolate this, we train the pixel side as RAE trains its decoder, only on noised encoder features and separately from the prior (its architecture and losses remain those of SemPix). We then either read the image out in a single step, i.e., in one forward pass from the semantic state as a decoder does, or sample it in 10 steps. Within a row, both numbers come from the same network and weights, so the comparison measures the effect of sampling the pixel side itself, independent of the decoder architecture or training loss.

Training1-step sampling10-step sampling
Independent (RAE-style)6.914.56
Coupled (CT)8.844.29

Under independent training, 10-step sampling lowers FID from 6.91 to 4.56. A generated semantic state does not determine a unique image, and generating the unspecified appearance step by step works better than completing it in one pass. We return to the second row in (ii).

AuthorsOpenReview ↗

Official Comment by Authors

(ii) Training on the prior's predictions mitigates the train–inference mismatch. In RAE, as in our independent setting, the pixel side is trained on real encoder features but receives generated states at inference. To examine the consequence, we compare the oracle FID, obtained by conditioning the pixel flow on real encoder features (the analogue of rFID), with the standard FID of sampled images:

Oracle FIDFID
Independent training (our RAE-style baseline)2.724.56
+ pixel flow trained on predicted states (detached)3.124.39
+ gradient feedback (CT)3.214.29

Our RAE-style baseline shows the reconstruction–generation trade-off known from latent diffusion (e.g., VA-VAE; Yao et al., 2025): it reconstructs best (oracle FID 2.72) but generates worst (4.56). The RAE paper also reports this mismatch (Zheng et al., 2025, Sec. 4.3 and Table 5): a decoder trained on clean encoder latents must decode generated latents that deviate from them, and adding noise to the decoder input improves gFID from 4.81 to 4.28 while worsening rFID from 0.49 to 0.57. As RAE notes, the noise also removes fine details. Our baseline already includes this augmentation, but noise only approximates the errors of the generator. Training the pixel flow on the semantic flow's predictions addresses the mismatch more directly and lowers FID from 4.56 to 4.39; gradient feedback lowers it further to 4.29, as analyzed in (iii). The slightly higher oracle FID does not affect generation, since at inference the pixel flow is always conditioned on states produced by the semantic flow, never on real encoder features.

This also explains the second row of the table in (i). Under independent training, the condition is a noised encoder feature of the target image, which already specifies much of the image, so a single step gives a reasonable result (6.91). Under CT, the condition is the semantic flow's prediction at a random noise level, ranging from detailed to nearly class-level, so the pixel flow learns to generate whatever the condition leaves open: a single step is worse (8.84), but 10-step sampling is best (4.29). CT thus makes the pixel flow rely more on its own sampling, and its benefit is realized together with the multi-step sampling of (i).

(iii) Gradient feedback improves the semantic flow itself. The feedback step lowers FID from 4.39 to 4.29. To confirm that this gain is robust, we repeated the comparison with three training seeds and six sampling seeds each; feedback gives lower FID in all 18 paired comparisons (mean 4.26 vs. 4.37). To locate where it acts, we cross-pair the CT model and the detached model at sampling time, taking the semantic flow from one and the pixel flow from the other. Both pixel flows are trained on predicted states in the same way; only in the CT model do image-loss gradients reach the semantic flow.

Semantic flow ↓ / Pixel flow →CT modelDetached model
CT model4.294.27
Detached model4.484.39

Within each column the pixel flow is fixed, and in both columns the semantic flow from the CT model is better (by 0.19 and 0.12). Trained with a feature-space loss alone, the semantic flow cannot tell which deviations matter for the image; the image-loss gradient supplies that signal. Notably, the semantic flow from the CT model helps even a pixel flow it was never trained with: paired with the detached model's pixel flow, it reaches 4.27, on par with its own jointly trained pixel flow (4.29). If the feedback merely co-adapted the two flows, this gain would disappear once the pixel flow is swapped. Instead, the feedback improves the sampled states themselves, and since both semantic flows target the same frozen DINOv3 space, the improvement transfers to another pixel flow.

AuthorsOpenReview ↗

Official Comment by Authors

Summary. Relative to RAE, SemPix (i) treats the pixel side as a conditional generator rather than a reconstruction decoder, (ii) trains it on the prior's own predictions instead of encoder features, which mitigates the train–inference mismatch more directly than RAE's noise augmentation, and (iii) feeds image-loss gradients back into the prior, all in a single training run without a separate decoder stage.

System level. For reference, we also report system-level results of RAE and RAEv2 (ImageNet 256×256, FID / IS; parameters include the separately trained decoder):

MethodParamsEpochsUnguidedGuided
RAE, DiT$^\text{DH}$-XL839M + 415M8001.51 / 242.91.13 / 262.6
RAEv2, DiT$^\text{DH}$-XL839M + 415M801.65 / 228.01.06 / 255.3
SemPix-L/16471M803.27 / 202.92.17 / 245.0
SemPix-H/16970M1602.40 / 225.21.83 / 250.4

A system-level comparison mixes several factors beyond the pixel-side design. RAE and RAEv2 use a larger generator (839M) together with a separately trained 415M decoder, and RAEv2 further aggregates features from multiple encoder layers, whereas SemPix's 471M covers both flows and requires no separate decoder stage. The controlled comparisons above, which fix all other factors as the reviewer suggested, isolate the effect of our design choices more directly.

Q3: V-Co

We thank the reviewer for bringing this recent and closely related work to our attention. The two designs differ in structure: V-Co co-denoises pixels and features in a single dual-stream network along a shared trajectory, whereas SemPix first generates the semantic state and then samples pixels with a separate, lightweight flow. Guided results on ImageNet 256×256:

ParamsEpochsFID / IS
SemPix-L/16471M802.17 / 245.0
V-Co-L/16918M2002.10 / 243.0
SemPix-H/16970M1601.83 / 250.4
V-Co-H/161.9B2001.85 / 246.5

At a similar or shorter training budget, SemPix matches V-Co with about half the parameters. Sampling also differs: V-Co uses 50-step Heun with CFG; SemPix uses 40 + 10 Euler steps.


We thank the reviewer again for the constructive comments, which have helped us clarify the positioning of SemPix. We hope the positioning and the new controlled comparisons address the reviewer's concerns. If so, we would be grateful if the reviewer would reconsider the assessment, in particular of informativeness. We are happy to discuss any remaining questions.

Reviewer_16xrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SemPix alleviates the learning burden of direct image generation by modeling semantic features and pixels separately. A semantic model first generates DINOv3 features that possess spatial structure; subsequently, a pixel model generates a complete image from independent noise. The core contribution of this paper lies in the coupled training of these two models: the pixel model uses the semantic model’s predictions as conditions, while the image generation loss is backpropagated through these conditions to update the semantic model. Simultaneously, feature supervision, which is provided by a fixed encoder, constrains the semantic predictions. In this way, the semantic model learns both the distribution of the pre-trained features and how to produce features that are more conducive to image generation. Furthermore, training requires only a single semantic prediction, eliminating the need to unroll the full sampling process.

Critical strengths
  1. This paper introduces a novel, explicit decomposition of pixel-space semantics and pixels. Unlike existing methods, which largely treat semantics as a condition, this approach generates the semantic states themselves via a flow matching.
  2. The coupled training scheme is clean and practical: a single clean semantic prediction conditions the pixel flow and remains attached to allow image-loss gradients to refine the semantic generator.
  3. This paper demonstrates that turning semantic structure into an explicit generative state reduces the optimization burden for native pixel diffusion.
Critical weaknesses
  1. Generating explicit semantic states and adapting them through image-generation feedback are the key contribution of this method. Existing ablations support a benefit to the complete system, but do not directly assess the quality of semantic states sampled from the learned prior.
  2. This method uses DINOv3, whereas the JiT+REPA baseline adopted from PixelGen uses DINOv2 representations. The comparisons retain the original supervision settings of the respective methods, and the paper provides neither a comparison using the same encoder nor an encoder-swap ablation.
Questions for authors
  1. The paper trains a separate Semantic flow but does not directly evaluate the quality of the generated semantic states. Could the authors provide additional experiments to assess its performance?
  2. this method uses DINOv3, while baselines (such as JiT+REPA) use DINOv2. Could the authors clarify how this difference affects the results and whether SemPix still achieves significant improvements when using the same pretrained visual encoder as the baselines?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank the reviewer for the positive assessment of the explicit semantic state and the coupled training scheme. Both questions are very helpful, and following them we have conducted two new sets of experiments.

W1 & Q1: Quality of the generated semantic states

We thank the reviewer for this valuable suggestion. Following it, we directly evaluated the states sampled from the semantic flow (SemPix-L/16, unguided, 50K samples), using real DINOv3 features as the reference. The sampled distribution approaches that of real features as training proceeds, and as conditions for the pixel flow, the sampled states cost only 0.63 FID relative to real features at 40 epochs and 0.21 at 80 epochs.

Semantic statesFD to real features ↓Image FID ↓
Sampled, 20 epochs28.578.19
Sampled, 40 epochs7.663.87
Sampled, 80 epochs5.383.27
Real, validation set (reference)3.26–

FD is the Fréchet distance in DINOv3 feature space, computed as in FID on token-averaged features, against training-set features. The reference row is the distance between real validation and real training features. Image FID uses the checkpoints in Fig. 4 of the paper.

Distribution distance. The distance to real features falls by 81% from 20 to 80 epochs, and its excess over the real-vs-real reference shrinks from 25.3 to 2.1.

Usefulness for image generation. For the 40- and 80-epoch models reported in the paper, we replace the sampled states with real encoder features as the pixel flow's condition (unguided image FID):

ModelSampled statesReal featuresGap
40 epochs3.873.240.63
80 epochs3.273.060.21

The gap is the cost of using sampled instead of real states: already small at 40 epochs, it shrinks by about two-thirds by 80 epochs. The semantic flow thus provides conditions close to real features, and keeps improving with training.

Effect of image-generation feedback on the states. The reviewer notes that the existing ablations show a benefit for the complete system but not for the semantic states themselves. To isolate the states, we cross-pair the two 40-epoch models of Table 3(b) of the paper (trained without Lmid\mathcal{L}_{\text{mid}}), which differ only in whether image-loss gradients reach the semantic flow (CT model) or are blocked by detaching the condition (detached model). We take the semantic flow from one and the pixel flow from the other (unguided FID):

Semantic flow ↓ / Pixel flow →CT modelDetached model
CT model4.294.27
Detached model4.484.39

Within each column the pixel flow is fixed, and in both columns the semantic flow from the CT model is better (by 0.19 and 0.12). Trained with a feature-space loss alone, the semantic flow cannot tell which deviations matter for the image; the image-loss gradient supplies that signal. Notably, the semantic flow from the CT model helps even a pixel flow it was never trained with: paired with the detached model's pixel flow, it reaches 4.27, on par with its own jointly trained pixel flow (4.29). If the feedback merely co-adapted the two flows, this gain would disappear once the pixel flow is swapped. Instead, the feedback improves the sampled states themselves, and since both semantic flows target the same frozen DINOv3 space, the improvement transfers to another pixel flow.

W2 & Q2: Encoder

We thank the reviewer for raising this point. To remove the encoder difference, we retrained SemPix-L/16 with DINOv2 ViT-B/14 (with registers), the encoder family used by the baselines, keeping the training recipe identical to the model in Table 1 of the paper.

EncoderFID, 40 epochs, unguided
SemPix-L/16 (paper)DINOv3 ViT-L/163.87
SemPix-L/16DINOv2 ViT-B/144.09
PixelGen-L/16DINOv2 ViT-B/147.53
JiT-L/16 + REPADINOv2 ViT-B/1423.67

With DINOv2-B, a smaller encoder than DINOv3-L, SemPix reaches 4.09, only 0.22 behind DINOv3-L and still 46% below the strongest baseline, PixelGen-L/16 (7.53). The advantage of SemPix therefore does not come from the choice of encoder.


We thank the reviewer again for the constructive questions, which have made our analysis more complete. We hope the new results address them, and if so, we would be grateful if the reviewer could take them into account in the final assessment.

Thank you again for your helpful feedback. In our response, we have added analyses of the generated semantic states and a comparison using DINOv2 to address your questions. When you have a chance, we would appreciate your thoughts on whether these results address your concerns. We would be happy to clarify any remaining points and continue the discussion.

Dear Reviewer 16xr,

Thank you again for your positive assessment. In our response above, we addressed your questions about the quality of the generated semantic states and about the encoder with new experiments. As the discussion period is ending, we would appreciate it if you could let us know whether these address your concerns and whether you have any other questions we can clarify. If your concerns have been addressed, we would be grateful if you could reflect this in your final assessment.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Author Responses and Discussion

We sincerely thank the AC and all reviewers for their time and valuable feedback. Below is a short summary of the discussion. The revised paper includes the results of our first response, with changes marked in blue.

Contributions acknowledged by the reviewers. SemPix models pretrained visual representations as an explicit generative state for pixel diffusion: a semantic flow generates dense semantic states, and a lightweight pixel flow synthesizes the image from them, with both trained end-to-end. The reviewers found this decomposition novel (16xr) and reasonable (vUHV), the jointly trained lightweight pixel flow an interesting design compared with RAE (EYus), and the coupled training scheme clean and practical (16xr). On ImageNet 256×256, SemPix converges substantially faster than pixel diffusion baselines, both without and with guidance.

Main concerns and our responses

  • Positioning w.r.t. RAE/RAEv2 (EYus; also vUHV). We clarified how SemPix differs from RAE/RAEv2 in its pixel side: a pixel flow that samples images and is trained jointly on the semantic flow's predictions, rather than a separately trained reconstruction decoder. A controlled comparison with everything else fixed shows that sampling the pixel side (6.91 → 4.56) and coupled training (CT, 4.56 → 4.29) each contribute.
  • Comparison with RAE decoder (EYus). Since SemPix does not use an adversarial loss, we retrained RAE's decoder with RAE's code, removing only its adversarial loss. On the same generated semantic states, the pixel flow gives lower FID than this decoder (4.09 vs. 5.79), also at matched compute (4.31), with less than a third of the parameters. We will add this comparison to the paper.
  • Contribution of the two-flow design (vUHV). The two-flow design is the first contribution of the paper, and CT trains it end-to-end. Against single-flow pixel diffusion with matched perceptual supervision (10.00), SemPix reaches 4.56 with independent training and 4.29 with CT.
  • Role of CT (vUHV, EYus). The gain is consistent across 3 training × 6 sampling seeds (lower FID in all 18 paired comparisons). Cross-pairing locates it in the semantic flow (0.12–0.19 FID), and oracle FID shows that CT adapts the pixel flow to the semantic flow's predictions. We clarified its role: it trains the pixel side for generation and adapts the semantic flow to it.
  • Confounds (vUHV: perceptual loss; 16xr: encoder). The gain from the semantic flow holds without any perceptual loss; Lperc\mathcal{L}_{\text{perc}} acts on the rendering quality of the pixel flow. With DINOv2-B, the baselines' encoder family, SemPix reaches 4.09 at 40 epochs vs. 7.53 for PixelGen-L/16.
  • Quality of the semantic states (16xr). The Fréchet distance to real DINOv3 features drops by 81% from 20 to 80 epochs; the FID cost of using sampled instead of real states is 0.63 at 40 epochs and 0.21 at 80 epochs.
  • Baselines and guided results (vUHV). We compared with RAE, RAEv2, CoReDi, PixelDiT, DeCo, PixelGen-XL and V-Co. With guidance at 40 epochs, SemPix reaches 2.99 vs. 4.95 for PixelDiT-L.

We have responded to all questions raised by the reviewers, including the follow-up questions from Reviewers EYus and vUHV during the discussion. vUHV indicated that most concerns have been addressed, and EYus indicated that they have no further questions. We would be grateful if the AC could take these responses into account in the final recommendation, and we are happy to provide any further information.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.