acceptodds
Under review as a conference paper at ICLR 2027

Extra Dimensions, Simpler Flows: Accelerating Flow Matching Training with Pretrained Visual Representations

16 views

Abstract

It is an effective strategy to accelerate the training of diffusion and flow-based generative models by jointly modeling image latent features and semantic representations extracted from pretrained vision encoders. However, stronger encoders do not always improve generation quality and can hinder optimization. We show that a semantic representation reduces velocity ambiguity only when it provides velocity-relevant information that is unavailable from the noisy image state, thereby distinguishing otherwise ambiguous velocity targets. Motivated by this insight, we propose Joint Semantic Flow Matching (JsFlow), which jointly transports the image state and semantic variables from multiple pretrained encoders. JsFlow accelerates semantic transport to provide more informative semantic states throughout the trajectory. Joint image–semantic prediction can still introduce gradient interference through the shared backbone. To address this issue, we predict semantic velocities through a separate branch that takes detached backbone features as input. This preserves semantic guidance during transport while preventing semantic-loss gradients from updating the image backbone. On ImageNet , JsFlow with LightningDiT-XL/1 achieves an unguided FID of after only K iterations. It outperforms SiT-XL/2 trained for M iterations (FID ) and LightningDiT-XL/1 trained for M iterations (FID ), requiring only and of their training iterations, respectively.

Primary area:
generative models
Keywords:
Flow Matching, Generative Modeling, Image Generation, Joint Semantic Flow Matching, Training Acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tSG5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Joint Semantic Flow Matching (JsFlow), which jointly generates image latents and semantic tokens from pretrained CLIP, DINOv1, and DINOv2 encoders. Its central argument is that semantic variables can reduce ambiguity in image-velocity prediction when they provide information unavailable from the intermediate image state. JsFlow completes semantic transport before image transport, allowing the generated semantic state to guide later image refinement. An enhanced variant uses a detached semantic prediction branch to limit gradient interference.

The authors prove a conditional-variance reduction result and evaluate the method on ImageNet 256×256 across LightningDiT and SiT backbones. On LightningDiT-XL/1, JsFlow reports an unguided FID of 5.91 after 50K iterations and a guided FID of 1.18 after 4M iterations. Ablations examine encoder choice, transport timing, and gradient decoupling.

Critical strengths
  1. Strong gains at matched training budgets. With LightningDiT-XL/1 at 50K iterations, JsFlow reduces unguided FID from REG’s 11.76 to 5.91. It reaches roughly REG’s 400K-iteration FID in about 100K iterations. This substantial early-training improvement is the strongest reason to consider acceptance.

  2. Evidence beyond one model configuration. Improvements are reported across LightningDiT model sizes and on SiT, with ablations supporting the choices of semantic features, transport timing, and gradient decoupling. This makes the result more compelling than a gain confined to one backbone.

Critical weaknesses
  1. The proposed mechanism is not directly validated. The conditional-variance result applies generally when a predictor receives additional information. The paper uses lower training MSE as a proxy for reduced irreducible velocity ambiguity, but MSE also reflects optimization and model error. This leaves the central explanation for JsFlow’s gains unconfirmed.

  2. The gains are insufficiently attributed to individual design choices. Comparisons with REG change the number of semantic tokens, transport schedule, and architecture together. The detached variant also adds about 100M parameters. Without controlled, capacity-matched ablations, it is difficult to tell which contributions are necessary for the reported improvement.

Questions for authors
  1. Clarification of existing evidence: Figure 6 uses image-velocity MSE to support the velocity-ambiguity explanation. What evidence separates a reduction in irreducible conditional variance from reductions in optimization or approximation error?

  2. Additional evidence that could change my assessment: Are there comparisons that isolate multiple semantic tokens, early semantic completion, and gradient decoupling under the same training setup, with a capacity-matched control for JsFlow†? If so, which component accounts for the largest gain?

8 comments
AuthorsOpenReview ↗

Response to Reviewer tSG5 part1,1/2

We sincerely thank the reviewer for the careful assessment and constructive suggestions.

Reply to W1 and Q1

[W1] The proposed mechanism is not directly validated. The conditional-variance result applies generally when a predictor receives additional information. The paper uses lower training MSE as a proxy for reduced irreducible velocity ambiguity, but MSE also reflects optimization and model error. This leaves the central explanation for JsFlow’s gains unconfirmed.

**[Q1]**Clarification of existing evidence: Figure 6 uses image-velocity MSE to support the velocity-ambiguity explanation. What evidence separates a reduction in irreducible conditional variance from reductions in optimization or approximation error?

We agree that training loss alone is an imperfect proxy for irreducible velocity ambiguity. However, directly estimating velocity variance in high dimensions is challenging [1,2], making training loss a practical indirect proxy. We assess its validity through theoretical and experimental evidence.

Using VarField to estimate velocity variance

  • Section D.3 introduces VarField, a neural variance estimator validated against analytic ground truth. Compared with self-normalized importance sampling (SNIS), it achieves substantially lower estimation errors, especially in high dimensions. A, G, and U denote distributions defined in Appendix D.3.
DimensionMethodA (%) ↓G (%) ↓U (%) ↓Mean (%) ↓
2SNIS27.870.2229.3119.13
VarField2.593.501.672.59
20SNIS41.148.5145.7131.79
VarField1.590.861.181.21
200SNIS62.3035.6863.1553.71
VarField5.744.178.035.98
400SNIS66.6944.0566.9959.24
VarField9.732.699.007.14
  • Section 4.4 uses VarField to estimate velocity variance across time for different encoders. The average estimates and generation results are reported below.
MethodVarField average variance ↓JsFlow† FID ↓
REPA0.737720.21
DINOv1-B0.71554.31
DINOv2-B0.72825.02
CLIP-B0.73547.38
MAE-L0.735419.40
MoCo-v3-B0.737018.22

DINOv1 achieves the lowest estimated variance and the best FID, followed by DINOv2. These measurements supplement the loss curves and support the connection between reduced prediction uncertainty and improved generation.

Assessing the Magnitude of Velocity Variance via Convergence Speed

  • Theorem 3 relates velocity variance to SGD convergence toward a stationary point, connecting variance analysis with optimization efficiency.
\leq \frac{2(\mathcal{L}(\theta_0)-\mathcal{L}^{\star})}{\eta T} + \frac{L_{\mathcal L}\eta}{B}(\sigma_{\mathrm{state}}^2+G^2\overline{\mathcal{V}}_Z).$$ This bound provides a theoretical connection between velocity variance and optimization efficiency. To examine this relationship experimentally, we measure the actual training steps required for image-branch MSE to reach ≤ 0.76 and compare them with the estimated variance across encoders. | Method | Actual Steps | VarField Average Variance ↓ | JsFlow† FID ↓ | |---|---:|---:|---:| | REPA | 120K | 0.7377 | 20.21 | | **DINOv1-B** | **50K** | **0.7155** | **4.31** | | DINOv2-B | 80K | 0.7282 | 5.02 | | CLIP-B | 90K | 0.7354 | 7.38 | | MAE-L | 100K | 0.7354 | 19.40 | | MoCo-v3-B | 110K | 0.7370 | 18.22 | The results across the six methods broadly show that **lower FID accompanies fewer training steps and lower estimated variance. Under comparable optimization conditions, convergence speed can serve as an approximate indicator of velocity variance.** Further details and figures are provided in the revised manuscript. [1] https://www.stochastik.math.uni-goettingen.de/preprints/B67.1%20(2005)%20019-041%20Munk.pdf [2] https://statistics.berkeley.edu/sites/default/files/tech-reports/696.pdf
AuthorsOpenReview ↗

Response to Reviewer tSG5 part2,2/2

Reply to W2 and Q2

[W2] The gains are insufficiently attributed to individual design choices. Comparisons with REG change the number of semantic tokens, transport schedule, and architecture together. The detached variant also adds about 100M parameters. Without controlled, capacity-matched ablations, it is difficult to tell which contributions are necessary for the reported improvement.

[Q2] Are there comparisons that isolate multiple semantic tokens, early semantic completion, and gradient decoupling under the same training setup, with a capacity-matched control for JsFlow†? If so, which component accounts for the largest gain?

To reduce the influence of architectural and parameter differences, all configurations use the same two-layer semantic-tower architecture as JsFlow†. The following ablations use LightningDiT-B/1(187M parameters), under the same training setup. Single-semantic configurations use DINOv2, while multi-semantic configurations combine CLIP, DINOv1, and DINOv2.

MethodFID ↓IS ↑sFID ↓
Single-semantic REG-style baseline15.4085.318.37
Single-semantic + temporal scheduling14.5789.338.32
Multi-semantic12.1091.366.35
Single-semantic + gradient decoupling14.5383.607.48
Multi-semantic + temporal scheduling9.68108.356.26
Multi-semantic + gradient decoupling11.7098.377.35
Single-semantic + temporal scheduling + gradient decoupling13.9490.657.54
All components9.64111.206.41

Multiple semantic tokens provide the largest individual FID improvement, reducing FID from 15.40 to 12.10. Temporal scheduling provides a further substantial improvement in the multi-semantic setting, reducing FID to 9.68. Combining all components achieves the best FID and IS.

Gradient decoupling serves a different purpose: semantic information reduces image velocity uncertainty, while gradient decoupling prevents semantic-loss interference with image-backbone optimization. It preserves semantic conditioning without allowing the semantic loss to directly update the image backbone. Together, these results clarify the complementary roles of semantic composition, temporal scheduling, and gradient decoupling.

To further examine whether the multi-semantic gain depends on encoder composition, we compare individual encoders and their subsets using LightningDiT-B/1 + JsFlow† without temporal scheduling.

EncoderSemantic tokensFID ↓IS ↑sFID ↓
CLIP115.8474.507.44
DINOv1112.2592.777.55
DINOv2114.5383.607.48
CLIP + DINOv1212.1294.077.55
DINOv1 + DINOv2212.2297.307.56
CLIP + DINOv2214.0984.687.71
CLIP + DINOv1 + DINOv2311.7098.377.35

The three-encoder combination achieves the best FID and IS. Removing any encoder worsens both metrics, confirming the value of complementary semantic information. However, the gains depend on the encoder combination and the information each encoder contributes.

Reviewer_tSG5OpenReview ↗

Clarification on VarField’s variance estimates

Thank you for the detailed response and the additional experiments. The shared-architecture ablations in Table 18 substantially address my concern about attributing the gains to individual components.

My remaining question concerns the interpretation of VarField. As acknowledged in Appendix D.2.2, Eq. (107), its estimate contains both the irreducible conditional variance and the frozen predictor’s squared error. The synthetic validation is helpful, but for the ImageNet results in Table 7, is there evidence that the predictor-error contribution is sufficiently small or comparable across encoders? Otherwise, differences in the estimates could still reflect differences in prediction accuracy rather than irreducible velocity ambiguity.

These additions improve my assessment of the paper. I would appreciate clarification on this remaining point before finalizing my rating.

Reviewer_hM9JOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper defines the difficulty of determining a target velocity using only noisy images in Flow Matching as "velocity ambiguity" and analyzes whether the incorporation of semantic representations can help reduce this uncertainty. Based on this, the author proposes JsFlow, which performs semantic generation first by jointly generating semantic tokens (derived from various pretrained vision encoders) and image latents. To mitigate gradient interference from the semantic loss, they also introduce JsFlow+, which employs a separate branch and a stop-gradient. Experiments on class-conditional image generation using the ImageNet-1K 256 dataset demonstrate improvements in generation quality.

Critical strengths
  • Analyzing the utility of semantic representations through the velocity ambiguity and linking this to gradient noise is a significant contribution. This provides a theoretical framework for understanding the conditions under which additional semantic information facilitates the training of flow matching model.

  • The experiments demonstrate performance improvements over baselines under sme conditions, as well as enhanced convergence across various backbones and model scales.s These findings substantiate the practical contributions regarding training efficiency.

Critical weaknesses
  • The experimental evidence for a reduction in velocity ambiguity is indirect. Since the MSE in the later stages of training encompasses both ambiguity and errors stemming from model approximation and optimization, it is insufficient to attribute observed performance improvements solely to a reduction in ambiguity.

  • There is a lack of controlled experiments to isolate the contributions of key design. It remains unclear whether the gains from the multi-encoder architecture arise from complementary information.

  • The experiments are limited to class-conditional generation on the ImageNet 256x256 dataset, which fails to sufficiently demonstrate the generalizability of the proposed method. In particular, it is unclear whether the utility of semantic tokens is specific to this experimental setup or if it persists in scenarios involving richer conditional information.

Questions for authors
  • Are there controlled comparisons of the number of tokens and computational cost between a single encoder and a multi-encoder? Could you discuss the differences and complementary information each encoder provides, and how combining them provides additional advantages in reducing velocity ambiguity?

  • I'm curious if the proposed method can be applied to text-to-image generation, in addition to class-conditional generation. It would be beneficial to include relevant experiments or a discussion on this.

5 comments
AuthorsOpenReview ↗

Response to Reviewer hM9J part1,1/3

We sincerely thank the reviewer for the thoughtful comments and constructive suggestions.

Reply to W1

[W1] The experimental evidence for a reduction in velocity ambiguity is indirect. Since the MSE in the later stages of training encompasses both ambiguity and errors stemming from model approximation and optimization, it is insufficient to attribute observed performance improvements solely to a reduction in ambiguity.

We agree that using training loss as a proxy for irreducible velocity ambiguity was simplistic ,However, directly estimating velocity variance in high-dimensional settings is challenging [1,2], making training loss a practical indirect proxy. To further justify this choice, we examine its validity from both theoretical and experimental perspectives.

Using VarField to estimate velocity variance

  • Section D.3 introduces VarField, a neural variance estimator, and validates it against analytic ground truth. Compared with conventional importance sampling (SNIS), VarField achieves substantially lower estimation errors, especially in high-dimensional settings(A, G, and U represent different distributions, with their detailed definitions provided in Appendix D.3.).
DimensionMethodA (%) ↓G (%) ↓U (%) ↓Mean (%) ↓
2SNIS27.870.2229.3119.13
VarField2.593.501.672.59
20SNIS41.148.5145.7131.79
VarField1.590.861.181.21
200SNIS62.3035.6863.1553.71
VarField5.744.178.035.98
400SNIS66.6944.0566.9959.24
VarField9.732.699.007.14
  • Section 4.4 uses VarField to estimate velocity variance across time for different encoders. The average estimates and corresponding generation results are reported below.
MethodVarField average variance ↓JsFlow† FID ↓
REPA0.737720.21
DINOv1-B0.71554.31
DINOv2-B0.72825.02
CLIP-B0.73547.38
MAE-L0.735419.40
MoCo-v3-B0.737018.22

DINOv1 achieves the lowest estimated variance and the best FID, followed by DINOv2. These measurements supplement the loss curves and support the connection between reduced prediction uncertainty and improved generation.

Assessing the Magnitude of Velocity Variance via Convergence Speed

  • Theorem 3 characterizes the relationship between variance and the rate at which SGD converges to a stationary point, connecting variance analysis with optimization efficiency.
\leq \frac{2(\mathcal{L}(\theta_0)-\mathcal{L}^{\star})}{\eta T} + \frac{L_{\mathcal L}\eta}{B}(\sigma_{\mathrm{state}}^2+G^2\overline{\mathcal{V}}_Z).$$ This bound directly relates the convergence rate of SGD toward a stationary point to variance, providing a theoretical basis for accelerating convergence through variance reduction. To further assess this relationship, we measure the actual training steps required for the image reconstruction MSE to reach ≤ 0.76. We use these steps to compare the relative magnitude of velocity variance across different encoders. | Method | Actual Steps | VarField Average Variance ↓ | JsFlow† FID ↓ | | ------------ | -------------- | --------------------------- | ------------- | | REPA | 120K | 0.7377 | 20.21 | | **DINOv1-B** | **50K** | **0.7155** | **4.31** | | DINOv2-B | 80K | 0.7282 | 5.02 | | CLIP-B | 90K | 0.7354 | 7.38 | | MAE-L | 100K | 0.7354 | 19.40 | | MoCo-v3-B | 110K | 0.7370 | 18.22 | The results show a consistent relationship that **lower fid is associated with fewer training steps, which correspond to lower velocity variance. Convergence speed can serve as an approximate indicator of the magnitude of velocity variance.**Further details and additional figures are provided in the revised manuscript. [1]https://www.stochastik.math.uni-goettingen.de/preprints/B67.1%20(2005)%20019-041%20Munk.pdf [2] [statistics.berkeley.edu/sites/default/files/tech-reports/696.pdf](https://statistics.berkeley.edu/sites/default/files/tech-reports/696.pdf)
AuthorsOpenReview ↗

Response to Reviewer hM9J part2,2/3

Reply to W2 and Q1

[W2] There is a lack of controlled experiments to isolate the contributions of key design. It remains unclear whether the gains from the multi-encoder architecture arise from complementary information.

[Q1] Are there controlled comparisons of the number of tokens and computational cost between a single encoder and a multi-encoder? Could you discuss the differences and complementary information each encoder provides, and how combining them provides additional advantages in reducing velocity ambiguity?

To reduce the influence of architectural and parameter differences, we use the same two-layer semantic-tower architecture as JsFlow†. The following ablations use LightningDiT-B/1(187 M,Single-semantic using Dinov2) under the same training setup and separately evaluate multiple semantic tokens, temporal scheduling, and gradient decoupling.

MethodFID ↓IS ↑sFID ↓
Single-semantic REG-style baseline15.4085.318.37
Single-semantic + temporal scheduling14.5789.338.32
Multi-semantic12.1091.366.35
Single-semantic + gradient decoupling14.5383.607.48
Multi-semantic + temporal scheduling9.68108.356.26
Multi-semantic + gradient decoupling11.7098.377.35
Single-semantic + temporal scheduling + gradient decoupling13.9490.657.54
All components9.64111.206.41

Multiple semantic tokens provide the largest individual FID improvement, reducing FID from 15.40 to 12.10. Temporal scheduling provides a further substantial improvement in the multi-semantic setting, reducing FID to 9.68. Combining all components achieves the best FID and IS.

The table below compares the computational costs and generation performance of single-encoder and multi-encoder configurations. Multi-encoder JsFlow increases parameters and FLOPs by only 1.03% and 1.01%, respectively, while reducing FID by 49.74% and improving IS by 89.73%.

Method#Params (M) ↓FLOPs (G) ↓FID ↓IS ↑
Single-encoder (REG)677238.211.7695.4
Multi-encoder (JsFlow)684240.65.91181.2
Increase (%)+1.03%+1.01%-49.74%+89.73%

We further compare individual encoders and their combinations using LightningDiT-B/1 + JsFlow† without temporal scheduling.

EncoderSemantic tokensFID ↓IS ↑sFID ↓
CLIP115.8474.507.44
DINOv1112.2592.777.55
DINOv2114.5383.607.48
CLIP + DINOv1212.1294.077.55
DINOv1 + DINOv2212.2297.307.56
CLIP + DINOv2214.0984.687.71
CLIP + DINOv1 + DINOv2311.7098.377.35

CLIP provides language-aligned information, while DINO encoders provide self-supervised visual representations. Removing any encoder from the three-encoder combination worsens FID and IS, supporting their complementary contributions.

AuthorsOpenReview ↗

Response to Reviewer hM9J part3,3/3

Reply to W3 and Q2

[W3] The experiments are limited to class-conditional generation on the ImageNet 256x256 dataset, which fails to sufficiently demonstrate the generalizability of the proposed method. In particular, it is unclear whether the utility of semantic tokens is specific to this experimental setup or if it persists in scenarios involving richer conditional information.

[Q2] I'm curious if the proposed method can be applied to text-to-image generation, in addition to class-conditional generation. It would be beneficial to include relevant experiments or a discussion on this.

We have extended JsFlow to text-to-image generation on CC3M using a text-conditioned MMDiT backbone. We use a 24-layer MMDiT backbone with a hidden width of 768, a frozen Stable Diffusion VAE, and a frozen CLIP-L text encoder. JsFlow additionally uses semantic features from CLIP, DINOv1, and DINOv2. Both REPA and JsFlow are trained with a batch size of 256 and evaluated at 50K and 100K training updates. Real CC3M image–text pairs are included as a reference.

MethodTraining updatesFID ↓IS ↑sFID ↓CLIPScore ↑
Real images—19.7527.3795.6123.77
REPA50K31.2014.5798.3017.27
JsFlow50K27.0017.17100.1718.54
REPA100K24.0615.9198.6718.40
JsFlow100K22.0420.0799.7719.74

JsFlow achieves lower FID and higher IS and CLIPScore at both checkpoints. At the same number of training updates, its FID is closer to the real-image reference, supporting faster progress toward the real distribution in this experiment. The higher CLIPScore also indicates better text–image alignment. These results provide evidence that JsFlow can extend beyond class-conditional generation to text-conditioned generation.

Reviewer_2gK6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies joint image–semantic flow matching through the conditional variance of image-velocity targets. It applies a conditional-variance decomposition to characterize when an additional semantic state reduces regression risk. The proposed method, JsFlow, jointly generates image latents and three global semantic tokens from CLIP, DINOv1, and DINOv2, completing semantic transport before image transport. An enhanced variant, JsFlow†, predicts semantic velocities with a separate branch that receives detached backbone features. On class-conditional ImageNet 256×256, standard JsFlow reaches an unguided FID of 5.91 after 50K iterations with LightningDiT-XL/1, and improvements are also reported across different model sizes and backbones (SiT). Longer training yields unguided/guided FIDs of 1.84/1.18.

Critical strengths
  1. On LightningDiT-XL/1 at 50K iterations, JsFlow reduces unguided FID from REG's 11.76 to 5.91, with only a small increase in backbone FLOPs. Appendix D.1 provides a particularly useful comparison: JsFlow at 100K reaches an FID of 3.25, comparable to REG's 3.30 at 400K. At 400K, the B/L/XL comparisons also consistently favor JsFlow: 11.05→7.26, 3.87→2.53, and 3.30→2.59 (Table 8). Figure 3 additionally compares REPA, REG, and SFD on two backbones. These results provide meaningful evidence of practical value.

  2. Standard JsFlow adds only three global semantic tokens and avoids training a separate semantic autoencoder. The reported parameter and FLOPs increases over REG are small. The appendix documents architectural choices, time conventions, encoder ablations, and results across model scales, providing useful practical information.

Critical weaknesses
  1. Latent Forcing [1] already jointly generates pretrained visual representations and images in separate branches. The supplied manuscript does not discuss its technical relationship to, or provide an empirical comparison with, the state-of-the-art Latent Forcing model, so the novelty is not fully established and the insights are not fully analyzed.

  2. The multi-encoder ablations do not establish a consistently substantial additional benefit: DINOv1→Multi-Enc improves FID from 5.43 to 4.89 for JsFlow, but only from 4.31 to 4.22 for JsFlow†, and from 13.31 to 13.20 in the no-REPA setting (Tables 5, 6, and 9). There is also no ablation that removes one encoder independently, which would help clarify the incremental information gained by adding multi-encoders one by one.

  3. The evaluation does not establish sufficient significance against current representation and generation systems. The encoder study is restricted to CLIP, DINOv1/v2, MAE, and MoCo-v3; it does not examine newer representation families such as DINOv3, SigLIP 2, SAM 3, or C-RADIOv4. Since the paper explores which representations simplify generation, it is important to test if a strong single encoder holds the best result or a combination of weak encoders also offer meaningful advantages.

  4. The final guided FID of 1.18 is worse than the best reported ImageNet 256×256 results of RAE (1.13) [2], RecTok (1.13) [3], REPA-E (1.12) [4], and VTP (1.11) [5]. The paper needs to demonstrate empirical superiority over these related works or explain how it is complementary to existing frameworks; otherwise, the current experimental results are insufficient to establish a consequential advance.

[1] https://arxiv.org/html/2602.11401v1#S3.SS2

[2] https://arxiv.org/html/2510.11690v1#S5.SS1

[3] https://arxiv.org/html/2512.13421v1#S4.SS2

[4] https://arxiv.org/html/2504.10483v3#S4.SS4

[5] https://arxiv.org/html/2512.13687v2#S6

Questions for authors
  1. What is the substantive methodological advance over Latent Forcing and over a single-encoder REG-style model with an accelerated semantic timestamp?

  2. For JsFlow†, a comparison of the same semantic tower with and without stop-gradient would also clarify whether gradient routing itself provides the claimed benefit.

  3. Does the multi-encoder advantage persist against a representative modern single encoder, such as DINOv3, SigLIP 2, or C-RADIOv4, under comparable feature-extraction and training budgets?

  4. Discuss the relationship to, and empirical comparison with, Latent Forcing and other state-of-the-art baselines: RAE (FID=1.13), RecTok (FID=1.13), REPA-E (FID=1.12), and VTP (FID=1.11).

3 comments
AuthorsOpenReview ↗

Response to Reviewer 2gK6 part1,1/3

We sincerely thank the reviewer for the thoughtful comments and constructive suggestions.

The original title may not fully clarify our main contributions and misguide you. Beyond improving generation performance and training speed, we seek to understand why modeling in a Semantic--Image Augmented Space accelerates training. Existing explanations based on models’ internal representations do not fully explain this acceleration [1]. Stronger representation capabilities do not necessarily lead to better generation quality, as shown in the table below.

EncoderImageNet Top-1 Acc. (%) ↑JsFlow FID ↓
REPA-20.21
CLIP75.099.76
DINOv177.865.43
DINOv283.946.43
MAE-L75.8020.93
MoCo-v372.8644.78

On SiT-B/2, better semantic representations do not always lead to better generation performance. For example, DINOv2 has higher ImageNet accuracy than DINOv1 (83.94% vs. 77.86%), but produces a worse FID (6.43 vs. 5.43). Moreover, using MoCo-v3 even degrades generation quality, resulting in an FID of 44.78.

We provide a new perspective on why modeling in a Semantic–Image Augmented Space accelerates training by analyzing the irreducible conditional variance of image FM targets. In our experiments, lower variance consistently accompanies faster convergence and better generation quality. These findings provide evidence for the acceleration mechanism and help guide the selection and design of semantic features. We have revised the manuscript to better highlight our contributions.

[1] [2507.01467v2] Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Think

Reply to W1 & Q1

[W1] Latent Forcing already jointly generates pretrained visual representations and images in separate branches. The supplied manuscript does not discuss its technical relationship to, or provide an empirical comparison with, the state-of-the-art Latent Forcing model, so the novelty is not fully established and the insights are not fully analyzed.

[Q1] What is the substantive methodological advance over Latent Forcing and over a single-encoder REG-style model with an accelerated semantic timestamp?

We thank the reviewer for highlighting Latent Forcing and related work. We agree that the original manuscript did not sufficiently clarify their technical relationships and differences. We have revised the Related Work section accordingly. Both Latent Forcing and REG explore the use of pretrained representations to improve image generation. Latent Forcing jointly models pretrained visual representations and images, while REG introduces representation conditioning into the generative process. These methods primarily explore effective designs through empirical comparisons and validate their benefits experimentally. Latent Forcing and JsFlow concrete implementations differ as follows.

AspectLatent ForcingJsFlow / JsFlow†
State spaceRaw pixels and spatial representation mapsImage latents and semantic tokens from one or multiple encoders
Condition injectionAdds aligned pixel and representation patch embeddingsProcesses semantic tokens alongside image tokens
Temporal scheduleSeparate representation and pixel timesAccelerates semantic completion along the joint transport path
Branch structureShared transformer with optional modality-specific output expertsJsFlow† uses a separate semantic tower
Gradient routingJoint pixel and representation objectivesJsFlow† blocks semantic-loss gradients from updating the image backbone

The corresponding FID results are reported in the table below. The methods use different model sizes, generation spaces, and training configurations. Although our reported result uses image latents, our formulation is compatible with pixel-space frameworks.

MethodParametersFID ↓
LF-DiT-L465M2.48
JsFlow (Ours)684M1.18

Moreover, our contribution goes beyond empirical design and evaluation. While Latent Forcing and REG emphasize empirical exploration, we use theoretical analysis to guide method design and validate the findings experimentally.

We use conditional velocity variance as a theoretical lens to explain how semantic representations improve image velocity prediction. By analyzing the irreducible conditional variance, we reveal why different representations lead to different generation performance and how semantic information reduces prediction uncertainty. Our theoretical analysis and experiments further establish connections between lower variance, faster convergence, and better generation quality. This provides a principled basis for understanding, evaluating, and selecting semantic representations, rather than relying solely on empirical design.

AuthorsOpenReview ↗

Response to Reviewer 2gK6 part2,2/3

Reply to W2 & W3 & Q3

[W2] The multi-encoder ablations do not establish a consistently substantial additional benefit: DINOv1→Multi-Enc improves FID from 5.43 to 4.89 for JsFlow, but only from 4.31 to 4.22 for JsFlow†, and from 13.31 to 13.20 in the no-REPA setting (Tables 5, 6, and 9). There is also no ablation that removes one encoder independently, which would help clarify the incremental information gained by adding multi-encoders one by one.

[W3] The evaluation does not establish sufficient significance against current representation and generation systems. The encoder study is restricted to CLIP, DINOv1/v2, MAE, and MoCo-v3; it does not examine newer representation families such as DINOv3, SigLIP 2, SAM 3, or C-RADIOv4. Since the paper explores which representations simplify generation, it is important to test if a strong single encoder holds the best result or a combination of weak encoders also offer meaningful advantages.

[Q3] Does the multi-encoder advantage persist against a representative modern single encoder, such as DINOv3, SigLIP 2, or C-RADIOv4, under comparable feature-extraction and training budgets?

We thank the reviewer for suggesting a broader encoder comparison. We have added DINOv3, SigLIP 2, and C-RADIOv4, together with multi-encoder combinations. To avoid confounding the comparison with the performance degradation previously observed when introducing the semantic loss, all experiments use LightningDiT-B/1 + JsFlow† without temporal scheduling.

EncoderImageNet Top-1 ↑ (CLS-token classification)FID ↓IS ↑sFID ↓
CLIP75.09%15.8474.507.44
DINOv177.86%12.2592.777.55
DINOv283.94%14.5383.607.48
DINOv382.48%15.0078.437.35
SigLIP 284.20%15.5076.517.23
C-RADIOv487.19%15.4276.747.11
MoCov372.86%17.4472.156.13
Haar2-stat12843.99%16.8170.657.63
CLIP + DINOv1—12.1294.077.55
DINOv1 + DINOv2—12.2297.307.56
CLIP + DINOv2—14.0984.687.71
CLIP + DINOv1 + DINOv2—11.7098.377.35
REPA baseline—17.7867.147.36

The results show that higher classification accuracy does not necessarily imply better generation quality. C-RADIOv4 achieves 87.19% Top-1 accuracy but an FID of 15.42, whereas DINOv1 achieves a lower FID of 12.25 despite its lower Top-1 accuracy of 77.86%.

More importantly, previous studies have largely focused on individual encoders, with limited exploration of combining complementary representations from multiple encoders. Empirical combinations of more advanced encoders or simply replacing a single encoder with a more powerful one do not necessarily lead to better results.. In contrast, combining complementary representations can yield better results. In particular, CLIP + DINOv1 + DINOv2 achieves the lowest FID of 11.70 and the highest IS of 98.37 among all tested configurations. This suggests that leveraging complementary semantic information is more beneficial than simply pursuing stronger individual encoders.

AuthorsOpenReview ↗

Response to Reviewer 2gK6 part3,3/3

Reply to Q2

[Q2] For JsFlow†, a comparison of the same semantic tower with and without stop-gradient would also clarify whether gradient routing itself provides the claimed benefit.

We thank the reviewer for the suggestion. To reduce the influence of architectural and parameter differences, we use the same two-layer semantic tower as JsFlow† and evaluate each design separately. The following ablations use LightningDiT-B/1.

MethodFID ↓IS ↑sFID ↓
Single-semantic REG-style baseline15.4085.318.37
Single-semantic + temporal scheduling14.5789.338.32
Multi-semantic12.1091.366.35
Single-semantic + gradient decoupling14.5383.607.48
Multi-semantic + temporal scheduling9.68108.356.26
Multi-semantic + gradient decoupling11.7098.377.35
Single-semantic + temporal scheduling + gradient decoupling13.9490.657.54
ALL9.64111.206.41

The table above supports our design choices. Our variance analysis shows that semantic conditioning reduces conditional velocity variance under the population-optimal predictor. However, this does not guarantee better generation with finite networks, as semantic-loss gradients can interfere with optimization. Stop-gradient stabilizes training and helps the method outperform REPA. Gradient decoupling improves FID by reducing semantic-loss interference while preserving semantic conditioning. Without it, some representations can cause severe training degradation. For instance, JsFlow with MoCo-v3 has an FID of 44.78 after 1M steps on SiT-B/2, indicating training failure.

EncoderImageNet Top-1 Acc. (%) ↑JsFlow FID ↓JsFlow† FID ↓
CLIP75.099.767.38
DINOv177.865.434.31
DINOv283.946.435.02
MAE-L75.8020.9319.40
MoCo-v372.8644.7818.22

Reply to W4 & Q4

[Q4] Discuss the relationship to, and empirical comparison with, Latent Forcing and other state-of-the-art baselines: RAE (FID=1.13), RecTok (FID=1.13), REPA-E (FID=1.12), and VTP (FID=1.11).

[W4] The final guided FID of 1.18 is worse than the best reported ImageNet 256×256 results of RAE (1.13) [2], RecTok (1.13) [3], REPA-E (1.12) [4], and VTP (1.11) [5]. The paper needs to demonstrate empirical superiority over these related works or explain how it is complementary to existing frameworks; otherwise, the current experimental results are insufficient to establish a consequential advance.

We have updated the Related Work section to clarify the technical relationships and differences. Given that 50K real validation images yield an FID of 1.68 against the reference statistics, while the model already achieves an FID of 1.18 with standard classifier-free guidance (CFG), we did not pursue finer-grained hyperparameter tuning or stronger guidance methods such as AutoGuidance. Our goal is not to achieve the best ImageNet FID across all existing generative frameworks, but to propose a variance-based design principle that can be integrated with different representations and generative frameworks.

Other methods improve generation through representation learning and tokenizer design. RAE uses pretrained representations as the generative space; RecTok introduces semantic distillation into flow trajectories; REPA-E jointly optimizes the VAE and diffusion model through representation alignment; and VTP combines contrastive learning, self-supervised learning, and reconstruction objectives for tokenizer pretraining. These directions complement our variance-based design principle. We have added compatibility experiments with configurations associated with these methods.

MethodGenerative-network parametersFID ↓IS ↑sFID ↓
REPA baseline137.90M17.7867.147.36
JsFlow†186.74M9.64111.206.41
VTP + JsFlow†186.79M10.11111.106.19
RecTok + JsFlow†186.89M8.05117.295.22
REPA-E + JsFlow†186.74M8.21112.554.78
RAE v1 + JsFlow†187.87M36.2151.5425.05
RAE v2 + JsFlow†121.67M5.90154.759.49

The combinations with RecTok, REPA-E, and RAE v2 achieve promising results, supporting the compatibility of JsFlow with different representations and generative configurations. RAE v1 performs poorly in our training setup, which may reflect limitations of the configuration used here. Switching to the more stable RAE v2 configuration substantially improves performance and yields largest gains over the JsFlow† baseline.

Comments on the paper

3 comments
Area_Chair_7SVzOpenReview ↗

Please engage in the discussion

Dear Reviewers 2gK6 and hM9J,

Thank you for your reviews and contributions so far. The authors have now posted their response, and I would like to kindly remind you to review it and engage in the discussion where appropriate.

Please also consider whether the authors' clarifications address any of your concerns and update your assessment if warranted.

Thank you for your time and effort.

Best regards,

Your AC.

AuthorsOpenReview ↗

Responses to Additional Potential Concerns 2gK6 part1,1/2

Dear Reviewer 2gK6:

We understand that reviewers may have many other responsibilities to attend to this week, and we fully appreciate that. To avoid leaving any potentially unresolved concerns unaddressed after the rebuttal period ends, when further author responses may no longer be possible, we would like to proactively clarify several additional points that might still raise questions or concerns for the reviewers.

The main contribution of our work goes beyond achieving faster training (up to 140×), as demonstrated by approaches such as REG and Latent Forcing. More fundamentally, we address a key question: Why do these additional representations actually help? We theoretically identify the reduction in conditional variance as the underlying mechanism.

Following the reviewers' suggestions, we have conducted additional experiments with more representation extractors, providing stronger evidence that representation quality and generative performance are not necessarily aligned: better representations do not always lead to better generation performance.

To better highlight this central contribution, we have reorganized the manuscript and added the following content about analysis of variance.

Using VarField to estimate velocity variance

  • Section D.3 introduces VarField, a neural variance estimator, and validates it against analytic ground truth. Compared with conventional importance sampling (SNIS), VarField achieves substantially lower estimation errors, especially in high-dimensional settings(A, G, and U represent different distributions, with their detailed definitions provided in Appendix D.3.).
DimensionMethodA (%) ↓G (%) ↓U (%) ↓Mean (%) ↓
2SNIS27.870.2229.3119.13
VarField2.593.501.672.59
20SNIS41.148.5145.7131.79
VarField1.590.861.181.21
200SNIS62.3035.6863.1553.71
VarField5.744.178.035.98
400SNIS66.6944.0566.9959.24
VarField9.732.699.007.14
  • Section 4.4 uses VarField to estimate velocity variance across time for different encoders. The average estimates and corresponding generation results are reported below.
MethodVarField average variance ↓JsFlow† FID ↓
REPA0.737720.21
DINOv1-B0.71554.31
DINOv2-B0.72825.02
CLIP-B0.73547.38
MAE-L0.735419.40
MoCo-v3-B0.737018.22

DINOv1 achieves the lowest estimated variance and the best FID, followed by DINOv2. These measurements supplement the loss curves and support the connection between reduced prediction uncertainty and improved generation.

As shown in Fig. 7, all estimated variance curves exhibit consistent values at the two endpoints, t=0t=0 and t=1t=1. Importantly, these endpoint values can be theoretically derived, providing two independent reference points for validating the accuracy of VarField.

Specifically, under the linear interpolation xt=(1−t)ϵ+tx1x_t=(1-t)\epsilon+tx_1, with independent standard Gaussian source noise ϵ∼N(0,Id)\epsilon\sim\mathcal{N}(0,I_d) and velocity target u=x1−ϵu=x_1-\epsilon, the conditional velocity variance at the noise endpoint (t=0t=0) is determined by the conditional data variance:

\frac{1}{d}\operatorname{tr}\left[ \operatorname{Cov}(u\mid x_0,z_0) \right]\= \frac{1}{d}\operatorname{tr}\left[ \operatorname{Cov}(x_1\mid z_0) \right]\ \approx 0.63. \end{aligned}$$ Here, the value of approximately $0.63$ is the average conditional data variance per dimension, consistently estimated across all evaluated methods. At the data endpoint ($t=1$), conditioning on the data endpoint and its semantic condition fixes $x_1$, while the source noise remains independent. Consequently, $$\begin{aligned} \frac{1}{d}\operatorname{tr}\left[ \operatorname{Cov}(u\mid x_1,z_1) \right]\= \frac{1}{d}\operatorname{tr}\left[ \operatorname{Cov}(\epsilon\mid x_1,z_1) \right]\= \frac{1}{d}\operatorname{tr}(I_d) =1. \end{aligned}$$ As demonstrated in Fig. 7, all methods recover these two theoretically determined endpoint values.
AuthorsOpenReview ↗

Responses to Additional Potential Concerns 2gK6 part2,2/2

Assessing the Magnitude of Velocity Variance via Convergence Speed

  • Theorem 3 characterizes the relationship between variance and the rate at which SGD converges to a stationary point, connecting variance analysis with optimization efficiency.
\leq \frac{2(\mathcal{L}(\theta_0)-\mathcal{L}^{\star})}{\eta T} + \frac{L_{\mathcal L}\eta}{B}(\sigma_{\mathrm{state}}^2+G^2\overline{\mathcal{V}}_Z).$$ This bound directly relates the convergence rate of SGD toward a stationary point to variance, providing a theoretical basis for accelerating convergence through variance reduction. To further assess this relationship, we measure the actual training steps required for the image reconstruction MSE to reach ≤ 0.76. We use these steps to compare the relative magnitude of velocity variance across different encoders. | Method | Actual Steps | VarField Average Variance ↓ | JsFlow† FID ↓ | | ------------ | -------------- | --------------------------- | ------------- | | REPA | 120K | 0.7377 | 20.21 | | **DINOv1-B** | **50K** | **0.7155** | **4.31** | | DINOv2-B | 80K | 0.7282 | 5.02 | | CLIP-B | 90K | 0.7354 | 7.38 | | MAE-L | 100K | 0.7354 | 19.40 | | MoCo-v3-B | 110K | 0.7370 | 18.22 | The results show a consistent relationship that lower fid is associated with fewer training steps, which correspond to lower velocity variance. Convergence speed can serve as an approximate indicator of the magnitude of velocity variance. Further details and additional figures are provided in the revised manuscript. ### Validating Conditional Variance Reduction Without Predictor Bias **Section D.2.4** introduces toy distributions with analytically available conditional means and covariances. The analytic results below evaluate conditional variance directly. The MC results average squared residuals against the **exact conditional mean**, rather than a learned predictor, and therefore contain no learned-predictor error. | Semantic dim. | 32D Reduction (Analytic, %) ↑ | 32D Reduction (MC, %) ↑ | 32D Cluster-KL ↓ | 128D Reduction (Analytic, %) ↑ | 128D Reduction (MC, %) ↑ | 128D Cluster-KL ↓ | |---|---:|---:|---:|---:|---:|---:| | 1 | 6.51 | 6.50 ± 0.03 | 8.329 ± 0.051 | 2.43 | 2.43 ± 0.03 | 39.665 ± 0.269 | | 2 | 13.19 | 13.19 ± 0.05 | 8.181 ± 0.059 | 4.93 | 4.93 ± 0.03 | 38.683 ± 0.419 | | 4 | 25.98 | 25.99 ± 0.05 | 4.365 ± 0.118 | 9.72 | 9.72 ± 0.04 | 40.504 ± 0.712 | | 8 | 30.85 | 30.86 ± 0.04 | 2.720 ± 0.039 | 11.54 | 11.55 ± 0.04 | 38.265 ± 0.282 | | 16 | **34.48** | **34.49 ± 0.05** | **2.471 ± 0.056** | **12.90** | **12.90 ± 0.04** | **16.776 ± 0.084** | MC results report mean ± standard deviation over 10 independent integrations. Cluster-KL reports mean ± standard error over 10 training seeds and is a diagnostic measure rather than the exact full-distribution KL. The analytic and MC results closely agree. Increasing the semantic dimension from 1 to 16 raises the analytic variance reduction from 6.51% to 34.48% on 32D data and from 2.43% to 12.90% on 128D data. The corresponding cluster-KL decreases from 8.329 to 2.471 and from 39.665 to 16.776. **These experiments directly demonstrate variance reduction without learned-predictor error and support its connection to improved distribution fitting.** Although these results do not directly establish that predictor-error differences are negligible in the ImageNet comparisons, they provide additional evidence that variance reduction can improve generation quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.