acceptodds
Withdrawn from ICLR 2027

Semantics as Source: Generative Flows from Semantic Embeddings Instead of Noise

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained semantic representations have become increasingly important in modern image generators, serving as conditioning signals, as feature-alignment targets, and even as the latent space itself. However, in every case the generator still begins from a content-free input, typically Gaussian noise. We propose semantics-as-source (SAS), which directly uses semantic embeddings as the source of a generative flow. The flow starts from the image's pooled semantic embedding, perturbed with noise, and the model learns the spatial residual that recovers the image's full latent token grid, which a frozen decoder maps to pixels. At inference, a small prior over embeddings supplies the starting point. Compared with conditioning and alignment on the same backbone and training budget, SAS gives better images and faster training across model scales and conditional and unconditional settings. On ImageNet-1k with SiT-XL/1, our source formulation achieves an unguided FID of 2.71 after only 40 epochs, compared with 2.96 for conditioning and 4.12 for alignment. SAS also roughly halves the expected squared distance the flow must cover, and the source route passes the Gaussian source's final 40-epoch FID within 10 epochs. Swapping only the source into RAE's tuned DiT-DH-XL recipe passes their released 80-epoch model by epoch 50 and beats it at an identical budget; with only 8 sampling steps our model matches the FID RAE reaches with 50 steps. With a decoupled head and autoguidance, SAS reaches FID 1.86 on ImageNet-1k after 160 epochs. These results suggest that semantic information is most effective not merely as guidance for a generative model, but as a structured starting point from which generation proceeds.

Primary area:
generative models
Keywords:
Image generation, Generative modeling, Flow matching

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_a7dpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses pooled semantic embeddings with noise as the starting point for image generation. Experimental results show that the proposed method achieves faster convergence and requires fewer sampling steps.

Critical strengths
  • The paper adds noise to the pooled features from a pre-trained model ( e.g., DINOv2) as the source of a generative flow, which eases the image generation.
Critical weaknesses
  • Closely related methods, such as CSFM, CrossFlow, LeDiFlow, and MM-FM, all use semantic embeddings to construct informative sources for easier image generation. The key contribution of the proposed method is not clear compared with related work. And there is no experimental comparison with closely related work.
  • The experiments are not convincing. 1) It's better to compare with related work using curves (performance over training epochs) and show how the methods converge (e.g., 800-epoch training curves compared with RAE, REPA, and some recent methods). 2) How the method performs on text-to-image generation is not clear.
Questions for authors
  • How is pϕ(e∣y)p_\phi(e\mid y) constructed and implemented for inference? Lines 202-205 briefly introduce it, but not very clear.
  • How diverse are the generated images, especially when starting from the same ee with different noise?
Reviewer_kc4HOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes semantics-as-source (SAS). In the RAE token space (DINOv2-B, x1∈R256×768x_1\in\mathbb{R}^{256\times768}), each training trajectory starts at z0=bc(e)+σεz_0=\mathrm{bc}(e)+\sigma\varepsilon, where e=1256∑jx1,je=\tfrac{1}{256}\sum_j x_{1,j} is the image's own pooled token embedding and σ2=(1−ρ)Eˉ\sigma^2=(1-\rho)\bar E matches the data energy. The flow learns the transport from this data-dependent source (an instance of the Albergo et al., 2023 coupled-interpolant framework) to the full token grid. At inference, an RCG-style MLP flow prior pϕ(e)p_\phi(e) or pϕ(e∣y)p_\phi(e\mid y) supplies ee.

The authors derive four simple identities. The source carries the global semantics. The expected transport energy is 2(1−ρ)Eˉ2(1-\rho)\bar E, roughly half that of an energy-matched Gaussian source. The spatial mean e+(1−t)σεˉe+(1-t)\sigma\bar\varepsilon is conserved along the path. Only the mean, and none of the residual, is given away.

Validation proceeds in three parts:

  1. Controlled comparison. SAS is compared against conditioning (COND, adaLN on the same prior embedding), alignment (ALIGN, REPA), and a Gaussian source (NOISE) on SiT-B/1 and SiT-XL/1, conditional and unconditional. The main setting is SiT-XL/1-cc at 40 epochs, with unguided FID-50K of 2.71, 2.96, 4.12, and 6.79 respectively.
  2. Mechanism analysis. This covers few-step sampling and chord-energy measurements, a shuffled-pair control, prior-vs-real-embedding checks, and a σ\sigma sweep.
  3. Transfer to RAE. Swapping the source into RAE's DiT-DH-XL recipe gives FID 2.18 → 2.03 at 80 epochs. With a DDT head, a learned residual source, and autoguidance, the model reaches 1.86 at 160 epochs.
Critical strengths

1. A well-controlled "where do semantics enter" comparison. The four routes share the encoder, latent space, backbone, data, and budget, and COND uses the same prior and embedding. This gives the cleanest evidence I know of that injecting ee as the source beats injecting it as an adaLN side input. The margin is modest but consistent across four settings (Table 1). This is the paper's main scientific contribution, and it is more informative than yet another SOTA claim.

2. Strong practical value at near-zero cost, especially for few-step sampling. The method changes one line of the sampler and adds no parameters to the flow. It roughly halves the chord energy, as predicted (2(1−ρ)Eˉ≈1.312(1-\rho)\bar E\approx1.31 vs. 2.60), and this translates into markedly better low-NFE behaviour. With 8 steps it matches RAE-80ep's 50-step FID, while the 800-epoch RAE model degrades sharply at 4–8 steps. The RAE drop-in result (2.18 → 2.03, with the pipeline checked against the released checkpoint) makes the method credible as a default.

Critical weaknesses

1. Novelty relative to conditional or informative sources, and missing direct baselines. Conditionally, SAS is p(x1∣e)p(x_1\mid e) flow matching from a mean-shifted Gaussian N(bc(e),σ2I)\mathcal N(\mathrm{bc}(e),\sigma^2 I), with ee sampled from an RCG prior. This is very close to the following prior work:

  • conditional or adaptive priors (Issachar et al. 2025; PriorGrad);
  • CAR-Flow's learned source shift;
  • CSFM;
  • and especially MM-FM, which also uses DINOv2-based modes coupled to targets in RAE latent space.

The distinguishing elements are the per-image (rather than per-class or per-mode) mean, combined with a learned prior over it. That is a reasonable but incremental design point. None of these methods is compared empirically, even though at least CAR-Flow-style shifts and MM-FM are directly implementable in the same space. Without these comparisons, the claim that "semantics should be the source" is supported only against conditioning, alignment, and the Gaussian source, not against the nearest competitors.

2. Attribution of the gains is not fully isolated.

(a) The shuffled-pair control is not a clean "geometry-only" control. Because the interpolant's spatial mean is (near-)deterministically readable, as identity (c) itself shows, a shuffled source feeds the network a confident but wrong semantic signal at every tt. It is therefore an adversarial input, not merely a reshaped p0p_0. Doing worse than NOISE is expected, and it does not show that the source shape is irrelevant. Two cleaner controls are possible:

  • a moment-matched Gaussian source with independent coupling;
  • input-level additive conditioning, i.e. a Gaussian source plus bc(e)\mathrm{bc}(e) injected through a separate input-embedding channel. This would separate "shorter transport" from "ee enters at the input rather than via adaLN".

(b) The RAE recipe experiments have no COND arm. SAS-on-DiT-DH adds a semantic-embedding prior, trained on 1.28M cached DINOv2 embeddings, that the released class-conditional RAE does not use. The 2.18 → 2.03 gain therefore conflates "source vs. noise" with "extra semantic conditioning via a learned prior". RCG showed that the latter alone substantially improves FID.

(c) Statistical support is thin. The SAS vs. COND gaps are small: 0.24–0.26 FID-10K at XL/1 in Table 1, and 2.71 vs. 2.96 in Table 2. Only NOISE is reported with multiple training seeds. Seed variance for SAS and COND is needed before calling the ordering robust.

A related issue concerns Table 4, which omits RAE's own longer-trained and guided results (to my knowledge substantially below 1.86 at 256²). The "1/9 of the updates" framing should be placed alongside these numbers.

Questions for authors
  1. Can you report SAS and COND with ≥3 training seeds in the main setting? Are the Table 1/2 gaps outside the seed spread?
  2. Can you add a COND arm (same prior, adaLN, Gaussian source) in the RAE DiT-DH-XL recipe? This would show how much of 2.18 → 2.03 comes from the source rather than from the embedding prior.
  3. Can you run a moment-matched Gaussian source and/or an input-additive conditioning baseline, to replace or complement the shuffled-pair control?
  4. Can you provide a direct comparison to MM-FM and to a CAR-Flow-style learned shift in the same RAE space?
  5. The class-conditional prior gives lower FID than real embeddings at all checkpoints (Sec. 4.5). Is the prior mode-seeking or low-temperature, in effect acting as implicit guidance? Please report prior-sample vs. real-embedding precision/recall in embedding space, and state whether "unguided" is still the appropriate label.

Clarifications:

  1. Real embeddings in Sec. 4.5: are they from training images or held-out images?
  2. Is the memorization check (I-JEPA nearest neighbour) also done for the class-conditional XL models and the 1.86 model?

Future work (does not affect my score):

  1. Does SAS transfer to non-semantic latents (SD-VAE), where ee must come from a separate encoder and the mean-pooling decomposition no longer holds?
  2. How does SAS behave with the coarse spatial-grid source suggested in Sec. 5?
Reviewer_jV5uOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes semantics-as-source (SAS), which starts a flow-matching generator in RAE[1] latent space from the image’s noised, mean-pooled-then-broadcasted DINOv2 embedding instead of pure noise. At inference, they first generate the embedding with a small residual-MLP generator trained with flow matching. They compare their method against conditioning on the embedding (RCG-like [2]) and aligning with the embedding during training (REPA-like[3]). As a drop-in for RAE’s Gaussian source, it improves the 80-epoch FID from 2.18 to 2.03.

[1] Diffusion Transformers with Representation Autoencoders. ICLR 2026 [2] Return of Unconditional Generation: A Self-supervised Representation Generation Method. Neurips 2024 [3] Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think. 2025

Critical strengths

A simple modification and the few-step results seem to be somewhat promising. SAS changes only the flow’s starting point and leaves the backbone unchanged; its one addition is a lightweight embedding prior. With 8 sampling steps, it matches the released model’s 50-step FID, and it degrades far less at low step counts than RAE.

Critical weaknesses

The central claim doesn't seem to be supported by the results. The paper’s main claim is that semantics are most effective as a structured starting point and not just as guidance. However, conditioning on the same embedding takes the main-setting FID from 6.79 to 2.96, making the embedding the source adds only 2.96 -> 2.71. The SAS-vs-COND gaps (3–11% FID-10K, as little as 0.24 points). Even if it holds for FID-50k and/or multiple seeds (3 runs), the improvement seems negligible. Further, it's unclear if the reported improvement over RAE is within training seed variance. The 0.15 gain is called “more than six times the generation-seed spread” by the authors, but the relevant variance is between training seeds.

No qualitative comparison. The paper doesn't provide a single generated image from the model in order to qualitatively compare the method and the competitors. Since the gains over the baselines are very tight, this makes it even more difficult to understand if the method results in any consistent improvements.

Questions for authors

Clarifications

  1. What is the training-seed variance for SAS and COND in the main setting?
  2. What sampler, step count, and temperature or guidance does the prior use?
  3. Are the prior’s sampling steps counted in the few-step NFE comparisons?
  4. Which REPA layer, loss weight, and projector does ALIGN use?
  5. All reported results are from early/mid training (160 epochs), have you tried training for longer

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.